RESTful Web API Design with Node.js 10

Third Edition

Learn to create robust RESTful web services with Node.js,

MongoDB, and Express.js

Valentin Bojinov

BIRMINGHAM - MUMBAI

RESTful Web API Design with

Node.js 10 Third Edition

form or by any means, without the prior written permission of the publisher, except in the case of brief

quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented.

However, the information contained in this book is sold without warranty, either express or implied. Neither the

author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged

to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products

mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the

accuracy of this information.

Commissioning Editor: Amarabha Banerjee

Acquisition Editor: Reshma Raman

Content Development Editor: Francis Carneiro

Technical Editor: Sachin Sunilkumar

Copy Editor: Shaila Kusanale

Project Coordinator: Sheejal Shah

Proofreader: Safis Editing

Indexer: Mariammal Chettiyar

Graphics: Jason Monteiro

Production Coordinator: Shraddha Falebhai

First published: October 2016

Second edition: October 2017

Third edition: April 2018

Production reference: 1300418

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78862-332-2

www.packtpub.com

mapt.io

Mapt is an online digital library that gives you full access to over

5,000 books and videos, as well as industry leading tools to help

you plan your personal development and advance your career. For

more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with

practical eBooks and Videos from over 4,000 industry

professionals

Improve your learning with Skill Plans built especially for

you

Get a free eBook or video every month

Mapt is fully searchable

Copy and paste, print, and bookmark content

PacktPub.com

Did you know that Packt offers eBook versions of every book

published, with PDF and ePub files available? You can upgrade to

the eBook version at www.PacktPub.com and as a print book customer,

you are entitled to a discount on the eBook copy. Get in touch with

us at service@packtpub.com for more details.

At www.PacktPub.com, you can also read a collection of free technical

articles, sign up for a range of free newsletters, and receive exclusive

discounts and offers on Packt books and eBooks.

Contributors

About the author

Valentin Bojinov studied computer science at the Technological

School of Electronic Systems in Sofia, Bulgaria, a college within the

Technical University of Sofia. He holds a B.Sc. in

telecommunication and information engineering. Valentin is an

expert in Java, SOAP, RESTful web services, and B2B integration.

He specializes B2B Integration and Service Oriented Architecture

and currently works as an Senior Integration Consultant in an UK

consultancy company Estafet Limited.

I would like thank my my dad Emil, for encouraging me to study programming almost

20 years ago, and to mummy Anka, for always being there for me! Special thanks to all

my mentors from school for showing me how to learn efficiently and to never give up. I

also have to mention my extraordinary schoolmates I had the chance to study with!

About the reviewers

Amit Kothari is a full-stack developer based in Melbourne,

Australia. He has more than 12 years experience in designing and

developing software systems and has worked on a wide range of

projects across various domains including telecommunication,

retails, banking and finance.

Amit is also the co-author of the book - Chatbots for eCommerce:

Learn how to build a virtual shopping assistant.

Erina has completed her master's and proactively working as an

assistant professor in the

computer science department of Thakur college, Mumbai. Her

enthusiasm in web

technologies inspires her to contribute for freelance JavaScript

projects, especially on

Node.js. Her research topics were SDN and IoT, which according to

her create amazing

solutions for various web technologies when they are used together.

Nowadays, she focuses on blockchain and enjoys fiddling with its

concepts in JavaScript.

What this book covers

Chapter 1, REST – What You Did Not Know, gives you a brief

introduction to the history of REST and how it couples with the

HTTP protocol.

Chapter 2, Getting Started with Node.js, teaches you how to install

Node.js and how to work with its package manager to install

modules. You'll also develop your first HTTP server application and

write automated unit tests for HTTP handler using mock request

objects.

Chapter 3, Building a Typical Web API, takes you through structuring

your application using human-readable URL and URI parameters.

You will get to develop a read-only RESTful service application,

using the filesystem for storage.

Chapter 4, Using NoSQL Databases, showcases how to use the

MongoDB NoSQL database, and explains the foundation of

document data stores.

Chapter 5, Restful API Design Guidelines, explains that there are a

number of prerequisites that a RESTful API should meet.

Chapter 6, Implementing a Full-Fledged RESTful Service, focuses on

implementing a production-ready RESTful service that uses NoSQL

to store its data. You will get to learn how to handle binary data and

how to version an API while it evolves.

Chapter 7, Preparing a RESTful API for Production, explains that

feature complete and full-fledged implementations aren't

necessarily production-ready.

Chapter 8, Consuming a RESTful API, showcases a sample frontend

client that serves as a consumption reference implementation.

Chapter 9, Securing the Application, covers restricting access to your

data by choosing an appropriate authentication approach. You'll

then be able to protect data leakage with transport layer security.

Packt is searching for authors

like you

If you're interested in becoming an author for Packt, please visit auth

ors.packtpub.com and apply today. We have worked with thousands of

developers and tech professionals, just like you, to help them share

their insight with the global tech community. You can make a

general application, apply for a specific hot topic that we are

recruiting an author for, or submit your own idea.

Table of Contents

Title Page

RESTful Web API Design with Node.js 10 Third Edition

Packt Upsell

Why subscribe?

PacktPub.com

Contributors

About the author

About the reviewers

Packt is searching for authors like you

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Conventions used

Get in touch

Reviews

1. REST – What You Did Not Know

REST fundamentals

Principle 1 – Everything is a resource

Principle 2 – Each resource is identifiable

by a unique identifier

Principle 3 – Manipulate resources via stand

ard HTTP methods

Principle 4 – Resources can have multiple re

presentations

Principle 5 – Communicate with resources in

a stateless manner

The REST goals

Separation of the representation and the resource

Visibility

Reliability

Scalability and performance

Working with WADL

Documenting RESTful APIs with Swagger

Taking advantage of the existing infrastructure

Summary

2. Getting Started with Node.js

Installing Node.js

Npm

Installing the Express framework and other modules

Setting up a development environment

Handling HTTP requests

Modularizing code

Testing Node.js

Working with mock objects

Deploying an application

Nodejitsu

Microsoft Azure

Heroku

Self-test questions

Summary

3. Building a Typical Web API

Specifying the API

Implementing routes

Querying the API using test data

Content negotiation

API versioning

Self-test questions

Summary

4. Using NoSQL Databases

MongoDB – a document store database

Database modeling with Mongoose

Testing a Mongoose model with Mocha

Creating a user-defined model around a Mongoose model

Wiring up a NoSQL database module to Express

Self-test questions

Summary

5. Restful API Design Guidelines

Endpoint URLs and HTTP status codes best practices

Extensibility and versioning

Linked data

Summary

6. Implementing a Full Fledged RESTful Service

Working with arbitrary data

Linking

Implementing paging and filtering

Caching

Supplying the Cache-Control header in Express applicatio

Discovering and exploring RESTful services

Summary

7. Preparing a RESTful API for Production

Documenting RESTful APIs

Testing RESTful APIs with Mocha

The microservices revolution

Summary

8. Consuming a RESTful API

Consuming RESTful services with jQuery

Troubleshooting and identifying problems on the wire

Cross Origin Resource Sharing

Content Delivery Networks

Handling HTTP status codes on the client side

Summary

9. Securing the Application

Authentication

Basic authentication

Passport

Passport's basic authentication strategy

Passport's OAuth Strategy

Passport's third-party authentication strategie

Authorization

Transport layer security

Self-test questions

Summary

Other Books You May Enjoy

Leave a review - let other readers know what you think

Preface

RESTful services have become the de facto standard data feed

providers for social services, news feeds, and mobile devices. They

deliver a large amount of data to millions of users. Thus, they need

to address high-availability requirements, such as reliability and

scalability. This book will show you how to utilize the Node.js

platform to implement a robust and performant data service. By the

end of this book, you will have learned how to implement a real-life

RESTful service, taking advantage of the modern NoSQL database

to serve both JSON and binary content.

Important topics, such as correct URI structuring and security

features, are also covered, with detailed examples, showing you

everything you need to know to start implementing the robust

RESTful APIs that serve content to your applications.

Who this book is for

This book targets developers who want to enrich their development

skills by learning how to develop scalable, server-side, RESTful

applications based on the Node.js platform. You also need to be

aware of HTTP communication concepts and should have a working

knowledge of the JavaScript language. Keep in mind that this is not

a book that will teach you how to program in JavaScript. Knowledge

of REST will be an added advantage but is definitely not a necessity.

To get the most out of this

book

1. Inform the reader of the things that they need to know

before they start, and spell out what knowledge you are

assuming

2. Any additional installation instructions and information

they need for getting set up

Download the example code

files

You can download the example code files for this book from your

account at www.packtpub.com. If you purchased this book elsewhere, you

can visit www.packtpub.com/support and register to have the files emailed

directly to you.

You can download the code files by following these steps:

1. Log in or register at www.packtpub.com.

2. Select the SUPPORT tab.

3. Click on Code Downloads & Errata.

4. Enter the name of the book in the Search box and follow the

onscreen instructions.

Once the file is downloaded, please make sure that you unzip or

extract the folder using the latest version of:

WinRAR/7-Zip for Windows

Zipeg/iZip/UnRarX for Mac

7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://githu

b.com/PacktPublishing/RESTful-Web-API-Design-with-Node.js-10-Third-Edition. In

case there's an update to the code, it will be updated on the existing

GitHub repository.

We also have other code bundles from our rich catalog of books and

videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

In this book, you will find a number of styles of text that distinguish

between different kinds of information. Here are some examples of

these styles, and an explanation of their meaning.

Code words in text, database tale names, folder names, filenames,

file extensions, pathnames, dummy URLs, user input, and Twitter

handles are shown as follows:

"This tells npm that our package depends on the URL and express

modules."

A block of code is set as follows:

router.get('/v1/item/:itemId', function(request, response, next) {

console.log(request.url + ' : querying for ' + request.params.itemId);

catalogV1.findItemById(request.params.itemId, response);

});

router.get('/v1/:categoryId', function(request, response, next) {

console.log(request.url + ' : querying for ' + request.params.categoryId);

catalogV1.findItemsByCategory(request.params.categoryId, response);

});

When we wish to draw your attention to a particular part of a code

block, the relevant lines or items are set in bold:

router.get('/v1/:categoryId', function(request, response, next) {

console.log(request.url + ' : querying for ' + request.params.categoryId);

catalogV1.findItemsByCategory(request.params.categoryId, response);

});

Any command-line input or output is written as follows:

$ npm install -g express

Bold: Indicates a new term, an important word, or words that you

see onscreen. For example, words in menus or dialog boxes appear

in the text like this. Here is an example:

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: Email feedback@packtpub.com and mention the book

title in the subject of your message. If you have questions about any

aspect of this book, please email us at questions@packtpub.com.

Errata: Although we have taken every care to ensure the accuracy

of our content, mistakes do happen. If you have found a mistake in

this book, we would be grateful if you would report this to us. Please

visit www.packtpub.com/submit-errata, selecting your book, clicking on the

Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any

form on the Internet, we would be grateful if you would provide us

with the location address or website name. Please contact us at

If you are interested in becoming an author: If there is a

topic that you have expertise in and you are interested in either

writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why

not leave a review on the site that you purchased it from? Potential

readers can then see and use your unbiased opinion to make

purchase decisions, we at Packt can understand what you think

about our products, and our authors can see your feedback on their

book. Thank you!

For more information about Packt, please visit packtpub.com.

REST – What You Did Not

Know

Over the last few years, we have started taking for granted that data

sources feeding for content, mobile device service feeds, or cloud

computing are all driven by modern technologies, such as RESTful

web services. Everybody has been talking about how their stateless

model allows applications to scale easily and how it emphasizes on

clear decoupling between data provisioning and data consumption.

Nowadays, architects have started introducing the concept of

microservices, aiming to reduce the complexity in systems by

splitting their core components into small independent pieces that

simply do a single task. So, enterprise-grade software is about to

become a composite of such microservices. This makes it easy to

maintain, and allows better life cycle management when new parts

need to be introduced. Unsurprisingly, most of the microservices

are serviced by RESTful frameworks. This fact may leave the

impression that REST was invented sometime in the last decade,

but that is far from the truth. In fact, REST has been around since

the last decade of the previous century!

This chapter will walk you through the foundation of

Representational State Transfer (REST) and will explain how

REST couples with the HTTP protocol. You will look into five key

principles that have to be considered while turning any HTTP

application into a RESTful service-enabled application. You will

also look at the differences between describing RESTful and

classic Simple Object Access Protocol (SOAP)-based web

services. Finally, you will learn how to utilize already-existing

infrastructure for your benefit.

In this chapter, we will cover the following topics:

REST fundamentals

REST with HTTP

Essential differences in the description, discovery, and

documentation of RESTful services compared to classical

SOAP-based services

Taking advantage of existing infrastructure

REST fundamentals

It actually happened back in 1999, when a request for comments

was submitted to the Internet Engineering Task Force (IETF;

http://www.ietf.org/) via RFC 2616: Hypertext Transfer Protocol-

HTTP/1.1. One of its authors, Roy Fielding, later defined a set of

principles built around the HTTP and URI standards. This gave

birth to REST as we know it today.

These definitions were given in https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_di

ssertation.pdf in the fifth chapter, Representational State Transfer (REST), of

Fielding's dissertation called Architectural Styles and the Design of Network-

Based Software Architectures. The dissertation is still available at

http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm.

Let's look at the key principles around the HTTP and URI

standards, sticking to which will make your HTTP application a

RESTful service-enabled application:

1. Everything is a resource

2. Each resource is identifiable by a unique identifier (URI)

3. Resources are manipulated via standard HTTP methods

4. Resources can have multiple representations

5. Communicate with resources in a stateless manner

Principle 1 – Everything is a

resource

To understand this principle, one must conceive of the idea of

representing data by a specific format and not by a physical file

containing a bunch of bytes. Each piece of data available on the

internet has a format that describes it, known as the content type;

for example, JPEG images, MPEG videos, HTML, XML, text

documents, and binary data are all resources with the following

content types: image/jpeg, video/mpeg, text/html, text/xml, and

application/octet-stream.

Principle 2 – Each resource is

identifiable by a unique

identifier

Since the internet contains so many different resources, they all

should be accessible via URIs and should be identified uniquely.

Furthermore, the URIs can be in a human-readable format, despite

the fact that their consumers are more likely to be software

programs rather than ordinary humans.

Human-readable URIs keep data self-descriptive and ease further

development against it. This helps you to keep the risk of logical

errors in your programs to a minimum.

Here are a few sample examples of such URIs representing different

resources in a catalog application:

http://www.mycatalog.com/categories/watches

http://www.mycatalog.com/categories/watches?collection=2018

http://www.mycatalog.com/categories/watches/model-xyz/image

http://www.mycatalog.com/categories/watches/model-xyz/video

http://www.mycatalog.com/archives/2017/categories/watches.zip

These human-readable URIs expose different types of resources in a

straightforward manner. In the preceding example URIs, it is quite

clear the data is items in a catalog, which are categorized watches.

The first link shows all the items in the category. The second shows

only the ones that are part of the 2018 collection. Next is a link

pointing to the image of the item, followed by a link to a sample

video. The last link points to a resource containing items from the

previous collection in a ZIP archive. The media types served per

each URI are rather easy to identify, with the assumption that the

data format of an item is either JSON or XML, so we can easily map

the media type of a self-described URL to one of the following:

JSON or XML documents describing the items

Images

Videos

Binary archive documents

Principle 3 – Manipulate

resources via standard HTTP

methods

The native HTTP protocol (RFC 2616) defines eight actions, also

known as HTTP verbs:

GET

POST

PUT

DELETE

HEAD

OPTIONS

TRACE

CONNECT

The first four of them just feel natural in the context of resources,

especially when defining actions for data manipulation. Let's make

a parallel with relative SQL databases where the native language for

data manipulation is CRUD (short for Create, Read, Update,

and Delete), originating from the different types of SQL

statements, INSERT, SELECT, UPDATE, and DELETE, respectively. In the same

manner, if you apply the REST principles correctly, the HTTP verbs

should be used as shown here:

Action

HTTP response status

code

Retrieves an existing

resource.

200 OK if the resource exists, 404

Not Found if it does not exist, and

500 Internal Server Error for other

errors.

Updates a resource. If the

resource does not exist, the

server can either decide to

create it with the provided

identifer or return the

appropriate status code.

200 OK if successfully updated,

201 Created if a new resource is

created, 404 Not found if the

resource to be updated does

not exist, and 500 Internal Server

Error for other unexpected

errors.

Creates a resource with an

identifier generated at

server side or updates a

resource with an existing

identifier provided from the

client. If this verb is to be

used only for creating but

not for updating, return the

appropriate status code.

201 CREATED if a new resource is

created, 200 OK if the resource

has been updated successfully,

409 Conflict if the resource

already exists and update is

not allowed, 404 Not Found if the

resource to be updated does

not exist, and 500 Internal Server

Error for other errors.

200 OK or 204 No Content if the

resource has been deleted

successfully, 404 Not Found if the

Deletes a resource. resource to be deleted does

not exist, and 500 Internal Server

Error for other errors.

Note that a resource might be created by either the POST or PUT HTTP

verbs, based on the policy of an application. However, if a resource

has to be created under a specific URI with an identifier provided

by the client, then PUT is the appropriate action:

PUT /categories/watches/model-abc HTTP/1.1

Content-Type: text/xml

Host: www.mycatalog.com

<?xml version="1.0" encoding="utf-8"?>

</Price></Price>

</Item>

HTTP/1.1 201 Created

Content-Type: text/xml

Location: http://www.mycatalog.com/categories/watches/model-abc

However, in your application, you may want to leave it up to the

backend RESTful service to decide where to expose the newly

created resource, and thus create it under an appropriate but still

unknown or non-existent location.

For instance, in our example, we might want the server to define the

identifier of newly created items. In such cases, just use the POST verb

to a URL without providing an identifier parameter. Then it is up to

the service itself to provide a new unique and valid identifier for the

new resource and to expose back this URL via the Location header of

the response:

POST /categories/watches HTTP/1.1

Content-Type: text/xml

Host: www.mycatalog.com

<?xml version="1.0" encoding="utf-8"?>

</Price></Price>

</Item>

HTTP/1.1 201 Created

Content-Type: text/xml

Location: http://www.mycatalog.com/categories/watches/model-abc

Principle 4 – Resources can

have multiple representations

A key feature of a resource is that it may be represented in a

different format from the one in which it is stored. Thus, it can be

requested or created in different representations. As long as the

specified format is supported, the REST-enabled endpoint should

use it. In the preceding example, we posted an XML representation

of a watch item, but if the server had supported the JSON format,

the following request would have been valid as well:

POST /categories/watches HTTP/1.1

Content-Type: application/json

Host: www.mycatalog.com

{

"watch": {

"id": ""watch-abc"",

"brand": "...",

"price": {

"-currency": "EUR",

"#text": "100"

}

HTTP/1.1 201 Created

Content-Type: application/json

Location: http://mycatalog.com/categories/watches/watch-abc

Principle 5 – Communicate

with resources in a stateless

manner

Resource manipulation operations through HTTP requests should

always be considered atomic. All modifications of a resource should

be carried out within an HTTP request in an isolated manner. After

the request execution, the resource is left in a final state; this

implicitly means that partial resource updates are not supported.

You should always send the complete state of the resource.

Back to our catalog example, updating the price field of a given item

would mean making a PUT request with a complete document

(JSON or XML) that contains the entire data, including the updated

price field. Posting only the updated price is not stateless, as it

implies that the application is aware that the resource has a price

field, that is, it knows its state.

Another requirement for your RESTful application to be stateless is

that once the service gets deployed on a production environment, it

is likely that incoming requests are served by a load balancer,

ensuring scalability and high availability. Once exposed via a load

balancer, the idea of keeping your application state at server side

gets compromised. This doesn't mean that you are not allowed to

keep the state of your application. It just means that you should

keep it in a RESTful way. For example, keep a part of the state

within the URI, or use HTTP headers to provide additional state-

related data

The statelessness of your RESTful API isolates the caller against

changes at the server side. Thus, the caller is not expected to

communicate with the same server in consecutive requests. This

allows easy application of changes within the server infrastructure,

such as adding or removing nodes.

Remember that it is your responsibility to keep your RESTful APIs stateless,

as the consumers of the APIs would expect them to be.

Now that you know that REST is around 18 years old, a sensible

question would be, "Why has it become so popular just quite

recently?" Well, we the developers usually reject simple,

straightforward approaches and, most of the time, prefer spending

more time on turning already-complex solutions into even more

complex and sophisticated ones.

Take classical SOAP web services, for example. Their various WS-*

specifications are so many, and sometimes so loosely defined, that

in order to make different solutions from different vendors

interoperable, a separate specification, WS-Basic Profile, has been

introduced. It defines extra interoperability rules in order to ensure

that all WS-* specifications in SOAP-based web services can work

together.

When it comes to transporting binary data with classical web

services over HTTP, things get even more complex, as SOAP-based

web services provide different ways of transporting binary data.

Each way is defined in other sets of specifications, such as SOAP

with Attachment References (SwaRef) and Message

Transmission Optimization Mechanism (MTOM). All this

complexity was caused mainly because the initial idea of the web

service was to execute business logic remotely, not to transport

large amounts of data.

The real world has shown us that, when it comes to data transfer,

things should not be that complex. This is where REST fits into the

big picture—by introducing the concept of resources and a standard

means for manipulating them.

The REST goals

Now that we've covered the main REST principles, it is time to dive

deeper into what can be achieved when they are followed:

Separation of the representation and the resource

Visibility

Reliability

Scalability

Performance

Separation of the

representation and the

resource

A resource is just a set of information, and as defined by principle 4,

it can have multiple representations; however, its state is atomic. It

is up to the caller to specify the desired media type with the Accept

header in the HTTP request, and then it is up to the server

application to handle the representation accordingly, returning the

appropriate content type of the resource together with a relevant

HTTP status code:

HTTP 200 OK in the case of success

HTTP 400 Bad Request if an unsupported format is given or for any

other invalid request information

HTTP 406 Not Acceptable if an unsupported media type is

requested

HTTP 500 Internal Server Error when something unexpected

happens during the request processing

Let's assume that, at server side, we have items resources stored in

an XML format. We can have an API that allows a consumer to

request the item resources in various formats, such as application/xml,

application/json, application/zip, application/octet-stream, and so on.

It would be up to the API itself to load the requested resource,

transform it into the requested type (for example, JSON or XML),

and either use ZIP to compress it or directly flush it to the HTTP

response output.

The caller would make use of the Accept HTTP header to specify the

media type of the response they expect. So, if we want to request

our item data inserted in the previous section in XML format, the

following request should be executed:

GET /category/watches/watch-abc HTTP/1.1

Host: my-computer-hostname

Accept: text/xml

HTTP/1.1 200 OK

Content-Type: text/xml

<?xml version="1.0" encoding="utf-8"?>

</Price></Price>

</Item>

To request the same item in JSON format, the Accept header needs to

be set to application/json:

GET /categoery/watches/watch-abc HTTP/1.1

Host: my-computer-hostname

Accept: application/json

HTTP/1.1 200 OK

Content-Type: application/json

{

"watch": {

"id": ""watch-abc"",

"brand": "...",

"price": {

"-currency": "EUR",

"#text": "100"

}

Visibility

REST is designed to be visible and simple. Visibility of the service

means that every aspect of it should self-descriptive and follow the

natural HTTP language according to principles 3, 4, and 5.

Visibility in the context of the outer world would mean that

monitoring applications would be interested only in the HTTP

communication between the REST service and the caller. Since the

requests and responses are stateless and atomic, nothing more is

needed to flow the behavior of the application and to understand

whether anything has gone wrong.

Remember that caching reduces the visibility of your RESTful applications

and in general should be avoided, unless needed for serving resources

subject to large amounts of callers. In such cases, caching may be an option,

after carefully evaluating the possible consequences of serving obsolete data.

Reliability

Before talking about reliability, we need to define which HTTP

methods are safe and which are idempotent in the REST context.

So, let's first define what safe and idempotent methods are:

An HTTP method is considered to be safe provided that,

when requested, it does not modify or cause any side effects

on the state of the resource

An HTTP method is considered to be idempotent if its

response stays the same, regardless of the number of times

it is requested, am idempotent request always gives back the

same request, if repeated identically.

The following table lists which HTTP methods are safe and which

are idempotent:

HTTP method Safe Idempotent

GET

Yes Yes

POST

No No

PUT

No Yes

DELETE

No Yes

Consumers should consider operation's safety and the idempotence

features in order to be served reliably.

Scalability and performance

So far, we stressed the importance of having stateless behavior for a

RESTful web application. The World Wide Web (WWW) is an

enormous universe, containing huge amount of data and a lot of

users, eager to get that data. The evolution of the WWW has

brought the requirement that applications should scale easily as

their load increases. Scaling applications that have a state is difficult

to achieve, especially when zero or close-to-zero operational

downtime is expected.

That's why staying stateless is crucial for any application that needs

to scale. In the best-case scenario, scaling your application would

require you to put another piece of hardware for a load balancer, or

bring another instance in your cloud environment. There would be

no need for the different nodes to sync between each other, as they

should not care about the state at all. Scalability is all about serving

all your clients in an acceptable amount of time. Its main idea is to

keep your application running and to prevent Denial of Service

(DoS) caused by a huge amount of incoming requests.

Scalability should not be confused with the performance of an

application. Performance is measured by the time needed for a

single request to be processed, not by the total number of requests

that the application can handle. The asynchronous non-blocking

architecture and event-driven design of Node.js make it a logical

choice for implementing an application that scales and performs

well.

Working with WADL

If you are familiar with SOAP web services, you may have heard of

the Web Service Definition Language (WSDL). It is an XML

description of the interface of the service and defines an endpoint

URL for invocation. It is mandatory for a SOAP web service to be

described by such a WSDL definition.

Similar to SOAP web services, RESTful services can also make use

of a description language, called WADL. WADL stands for Web

Application Definition Language. Unlike WSDL for SOAP web

services, a WADL description of a RESTful service is optional, that

is, consuming the service has nothing to do with its description.

Here is a sample part of a WADL file that describes the GET operation

of our catalog service:

<?xml version="1.0" encoding="UTF-8"?>

<application xmlns="http://wadl.dev.java.net/2009/02"

xmlns:service="http://localhost:8080/catalog/"

xmlns:xsd="http://www.w3.org/2001/XMLSchema">

</grammer>

</request>

<representation mediaType="application/xml" element="service:item"

</response>

<representation mediaType="application/xml" element="service:item"

</response>

</method>

</resource>

</resources>

</application>

This extract of a WADL file shows how application, exposing

resources is described. Briefly, each resource must be part of an

application. The resource provides a, where it is located with the base

attribute, and describes each of its supported HTTP methods in a

method. Additionally, an optional doc element can be used at

resource and application to provide additional documentation

about the service and its operations.

Though WADL is optional, it significantly reduces the efforts of

discovering RESTful services.

Documenting RESTful APIs

with Swagger

Public APIs exposed on the web should be well documented,

otherwise it would be difficult for developers to use them in their

applications. While WADL definitions might be considered a source

of documentation, they address a different problem—the discovery

of the service. They serve metadata for the services to machines, not

to humans. The Swagger project (https://swagger.io/) addresses the

need for neat documentation of RESTful APIs. It defines a meta

description of an API from an almost human-readable JSON

format. The following is a sample swagger.json file, partially describing

the catalog service:

{

"swagger": "2.0",

"info": {

"title": "Catalog API Documentation",

"version": "v1"

"paths": {

"/categories/{id}" : {

"get": {

"operationId": "getCategoryV1",

"summary": "Get a specific category ",

"produces": [

"application/json"

"responses": {

"200": {

"description": "200 OK",

"examples":

{"application/json": {

"id": 1,

"name": "Watches",

"itemsCount": 550

}

"404": {"description" : "404 Not Found"},

"500": {"description": "500 Internal Server Error"}

}

"consumes": ["application/json"]

}

The swagger.json file is really straightforward: it defines a name and

version of your API and gives a brief description of each operation it

exposes, nicely coupled with a sample payload. But the real benefit

from it comes in another subproject of Swagger, called swagger-ui (http

s://swagger.io/swagger-ui/), which actually renders this data

from swagger.json nicely into an interactive web page that only

provides documentation, but also allows interaction with the

service:

We will have a look at and utilize the swagger-ui Node.js module to

provide the API that we will develop later in the book, with up-to-

date documentation.

Taking advantage of the

existing infrastructure

The best part of developing and distributing RESTful applications is

that the infrastructure needed is already out there, available to you.

As RESTful applications use the existing web space heavily, you

need to do nothing more than follow the REST principles when

developing. In addition, there are plenty of libraries available out

there for any platform, and I do mean any platform. This eases the

development of RESTful applications, so you just need to choose

your preferred platform and start developing.

Summary

In this chapter, you learned about foundation of a REST, looking at

five key principles that transform a web application into a REST-

enabled application. We made a brief comparison between RESTful

services and classical SOAP web services, and finally took a look at

how RESTful services are documented and how we can simplify the

discovery of the services we develop.

Now that you know the basics, we are ready to dive into the Node.js

way of implementing RESTful services. In the next chapter, you will

learn about the essentials of Node.js and the accompanying tools

that it is necessary to use and understand in order to build a real-

life fully-fledged web service.

Getting Started with Node.js

In this chapter, you will gain your first real Node.js experience. We

will start by installing Node.js, along with some modules we will use

throughout this book. Then, we will set up a development

environment. Throughout the book, the Atom IDE will be used. Yes,

GitHub's online editor has finally landed for desktop environment

and is available on a platform of your preference!

Next, we will create a workspace and start developing our first

Node.js application. It will be a simple server application processing

incoming HTTP requests. We will go one step further,

demonstrating how to modularize and unit test our JavaScript code.

Finally, we will deploy our first application on the Heroku Cloud

Application platform.

To sum up, in this chapter, we will cover the following topics:

Installing Node.js

Installing the Express framework and other modules

Setting up a development environment

Handling HTTP requests

Modularizing code

Testing Node.js

Deploying an application

Installing Node.js

Let's start our journey through Node.js with a Node.js installation.

Installers are available for both Windows and macOS at https://nodejs

.org/en/download/. At the time of writing, Node.js 10 has just been

released as the current version, and is about to become the next

Long Term Support version, in August 2018. Linux users can either

build Node.js from the available Linux binaries or make use of their

package manager, as Node.js is available with most of the popular

package repositories for different Linux distributions. For instance,

Ubuntu and other Debian-based distributions should first point to

the latest Node.js 10 package and then install via the apt-get

command from the shell:

curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -

sudo apt-get install nodejs

If you decide to go with an installer available for macOS or

Windows, a wizard will guide you through a rather typical

installation procedure, where you will have to accept the Node.js

license agreement and then provide an installation path.

Linux users performing installations via package managers will

need to install Node Package Manager (npm) separately; we

will do that in the next section.

After a successful installation, you should have Node set on

your PATH environment variable.

The installer will have preselected for you the Node.js runtime,

npm, shortcuts to online documentation resources, as well as the

option to add Node.js and npm to your operating system PATH

environment variable.

To verify that your installation was successful, execute the following

from your shell:

node --version

At the time of writing, the latest Node.js version is 10.0.0 so, as expected, this

version number will be the output of the version check. Node.js 10 will be the

next Long Term Supported version, so it will stay actual for the next few

years

Npm

Node.js eases support to third-party open source-developed

modules by providing npm. It allows you, as a developer, to easily

install, manage, and even provide your own module packages. The

npm package repository is available at http://www.npmjs.org/ and is

accessible via its command-line interface.

If you didn't use the installer, then you will need to install npm

separately. For example, Ubuntu users can make use of their

package installer as follows:

apt-get npm install

If you upgraded your Node.js installation and you had previously

installed npm 5.6, you would be asked to upgrade it to version 6. To

do that, just execute:

sudo npm i -g npm

Once npm is installed, it is useful to set it permanently in your user

profile's PATH environment variable by editing the ~/.profile file to

export the path to npm as follows:

export PATH=$PATH:/path/to/npm

After a successful npm installation, use npm's ls option to display

the currently-installed Node.js modules:

bojinov@developer-machine:~$ npm ls

/home/bojinov

├─┬ accepts@1.3.3

│ ├─┬ mime-types@2.1.13

│ │ └── mime-db@1.25.0

│ └── negotiator@0.6.1

├── array-flatten@1.1.1

├─┬ cache-control@1.0.3

│ ├─┬ cache-header@1.0.3

│ │ ├── lodash.isnumber@2.4.1 deduped

│ │ ├── lodash.isstring@2.4.1

│ │ └── regular@0.1.6 deduped

│ ├─┬ fast-url-parser@1.1.3

│ │ └── punycode@1.4.1

│ ├─┬ glob-slasher@1.0.1

│ │ ├── glob-slash@1.0.0

│ │ ├─┬ lodash.isobject@2.4.1

│ │ │ └── lodash._objecttypes@2.4.1

│ │ └─┬ toxic@1.0.0

│ │ └── lodash@2.4.2

│ ├─┬ globject@1.0.1

│ │ └── minimatch@2.0.10 extraneous

│ ├── lodash.isnumber@2.4.1

│ ├── on-headers@1.0.1

│ └── regular@0.1.6

├── content-disposition@0.5.1

├── content-type@1.0.2

├── cookie@0.3.1

├── cookie-signature@1.0.6

Installing the Express

framework and other modules

Now that we have npm installed, let's make use of it and install some

of the modules we will be using heavily throughout this book. The

most important among them is the Express framework

(http://www.expressjs.com/). It is a flexible web application framework for

Node.js, providing a robust RESTful API for developing single or

multi-page web applications. The following command will

download the Express module from the npm repository and make it

available for our local Node.js installation:

npm install -g express

You will find the express module among the results of an npm ls after a

successful installation. Later in this chapter, we will learn how to

write unit tests for our Node.js modules. We will need the nodeunit

module for that purpose:

npm install nodeunit -g

The -g option will install nodeunit globally. This means that the

module will be stored at a central place on your filesystem; usually,

that is either /usr/lib/node_modules or /usr/lib/node, but that can be

configured to the global configuration of your Node.js. Globally

installed modules are available to all running node applications.

Locally installed modules will be stored in a node_modules subdirectory

of the current working directory of your project and will be available

only to that single project.

Now, coming back to the nodeunit module—it provides basic assert

test functions for creating basic unit tests as well as tools for

executing them.

Before starting to develop with Node.js, we have one more thing to

look into: the package descriptor file of a Node.js application.

All Node.js applications or modules contain a package.json descriptor

file. It provides meta-information about the module, its authors,

and the dependencies it uses. Let's take a look at the package.json

file of the express module we installed earlier:

{

"_from": "express",

"_id": "express@4.16.1",

"_inBundle": false,

"_integrity": "sha512-

STB7LZ4N0L+81FJHGla2oboUHTk4PaN1RsOkoRh9OSeEKylvF5hwKYVX1xCLFaCT7MD0BNG/gX2WF

MLqY6EMBw==",

"_location": "/express",

"_phantomChildren": {},

"_requested": {

"type": "tag", "registry": true, "raw": "express", "name": "express",

"escapedName": "express","rawSpec": "", "saveSpec": null, "fetchSpec":

"latest"

"_requiredBy": [

"#USER"

"_resolved": "https://registry.npmjs.org/express/-/express-4.16.1.tgz",

"_shasum": "6b33b560183c9b253b7b62144df33a4654ac9ed0",

"_spec": "express",

"_where": "/home/valio/Downloads",

"author": {

"name": "TJ Holowaychuk",

"email": "tj@vision-media.ca"

"bugs": {

"url": "https://github.com/expressjs/express/issues"

"bundleDependencies": false,

"contributors": [

{

"name": "Aaron Heckmann",

"email": "aaron.heckmann+github@gmail.com"

...,

{

"name": "Young Jae Sim",

"email": "hanul@hanul.me"

}

"dependencies": {

"accepts": "~1.3.4",

"array-flatten": "1.1.1",

"body-parser": "1.18.2",

...,

"type-is": "~1.6.15",

"utils-merge": "1.0.1",

"vary": "~1.1.2"

"deprecated": false,

"description": "Fast, unopinionated, minimalist web framework",

"devDependencies": {

"after": "0.8.2",

"connect-redis": "~2.4.1",

...,

"should": "13.1.0",

"supertest": "1.2.0",

"vhost": "~3.0.2"

"engines": {

"node": ">= 0.10.0"

"files": ["LICENSE", "History.md", "Readme.md", "index.js","lib/"],

"homepage": "http://expressjs.com/",

"keywords": [

"express", "framework", "sinatra", "web", "rest", "restful", "router",

"app", "api"

"license": "MIT",

"name": "express",

"repository": {

"type": "git",

"url": "git+https://github.com/expressjs/express.git"

"scripts": {

"lint": "eslint .",

"test": "mocha --require test/support/env --reporter spec --bail --check-

leaks test/ test/acceptance/"

"version": "4.16.1"

}

The name and the version of the package are mandatory properties

for every module. All other pieces of meta-information, such as the

contributors list, repository type and location, license information,

and so on, are optional. One of the most interesting properties,

which is worth mentioning, is the dependencies property. It tells npm

which modules your package depends on. Let's take a deeper look at

how this is specified. Each dependency has a name and a version.

This tells npm that the package depends on the accepts module with

version 1.3.4 and the body-parse module with version 1.8.2. So,

when npm installs the module, it will implicitly download and

install the latest minor versions of the dependencies, in case they

are not already available.

The version of a dependency is specified in the following format:

major.minor.patch-version. You can specify npm if you want npm to use

exactly the specified version, or you can have npm always download

the latest available minor version, by starting the version with ~; see

the accepts dependency for reference.

For more information on versioning, visit the website of the semantic

versioning specification at http://www.semver.org/.

Depending on automatically managed version may result in backward

incompatibility, make sure you test your application each time you switch a

version.

Setting up a development

environment

JavaScript developers are rarely used to developing their projects in

an IDE; most of them use text editors and tend to be prejudiced

against anything that contradicts their views. GitHub has finally

managed to calm most of them down by releasing the Atom IDE for

the desktop environment. This may not solve all of the arguments

about which is the best environment, but will at least bring some

peace, and let people concentrate on their code, not on the tooling,

which in the end is a matter of personal preference. The samples in

this book are developed in the Atom IDE, but feel free to use any

piece of software that can create files, including command-line

editors such as vi or vim, if that would make you feel like a JS

superhero, though bear in mind that superheroes are so 20

century!

You can download the Atom IDE from https://ide.atom.io/.

It is time to start our first Node.js application, a simple web server

responding with Hello from Node.js. Select a directory from your

project, for example, hello-node, then open a shell Terminal from it

and execute npm init:

npm init

package name: (hello-node)

version: (1.0.0)

description: Simple hello world http handler

entry point: (index.js) app.js

test command: test

git repository:

keywords:

author: Valentin Bojinov

license: (ISC)

About to write to /home/valio/nodejs8/hello-node/package.json:

{

"name": "hello-node",

"version": "1.0.0",

"description": "Simple hello world http handler",

"main": "app.js",

"scripts": {

"test": "test"

"author": "Valentin Bojinov",

"license": "ISC"

}

Is this ok? (yes) yes

A command-line interaction wizard will ask you for your project

name, its version, as well as some other metadata such as Git

repository, your name, and so on, and will finally preview the

package.json file it is to generate; when complete, your first Node.js

project is ready to begin.

Now is the appropriate time to spend some time on the code

convention used in this book; ES6 inline anonymous functions will

be used when short callback functions should be defined, while

regular javascript function will be used when reusability and

testability is expected.

Start the Atom IDE, select File | Add Project Folder..., and import

the directory you defined the project in. Finally, after a successful

import, you will see the generated package.json file in the project.

Right-click on the directory, select New File, and create a file called

hello-node.js:

var http = require('http');

http.createServer((request, response) => {

response.writeHead(200, {

'Content-Type' : 'text/plain'

});

response.end('Hello from Node.JS');

console.log('Hello handler requested');

}).listen(8180, '127.0.0.1', () => {

console.log('Started Node.js http server at http://127.0.0.1:8180');

});

The hello-node.js file uses the Node.js HTTP module to start listening

for incoming requests on port 8180. It will reply with static Hello from

Node.JS to each request and will log a hello log entry in the console.

Before starting the application, we have to install the http module

that creates an HTTP server for it. Let's install it globally together

with the --save option, which will add a dependency to it in the

package.json file of the project. Then we can start the app:

npm install -g http --save

node hello-node.js

Opening http://localhost:8180/ from your browser will result in sending

a request to the server application, which will make a log entry in

the console and will output Hello from Node.JS in your browser:

Handling HTTP requests

Currently, our server application will behave in the same way, no

matter what kind of HTTP request is processed. Let's extend it in

such a way that it behaves more like an HTTP server, and start

differentiating the incoming requests based on their type, by

implementing handler functions for each type of request.

Let's create a new hello-node-http-server.js as follows:

var http = require('http');

var port = 8180;

function handleGetRequest(response) {

response.writeHead(200, {'Content-Type' : 'text/plain'});

response.end('Get action was requested');

}

function handlePostRequest(response) {

response.writeHead(200, {'Content-Type' : 'text/plain'});

response.end('Post action was requested');

}

function handlePutRequest(response) {

response.writeHead(200, {'Content-Type' : 'text/plain'});

response.end('Put action was requested');

}

function handleDeleteRequest(response) {

response.writeHead(200, {'Content-Type' : 'text/plain'});

response.end('Delete action was requested');

}

function handleBadRequest(response) {

console.log('Unsupported http mehtod');

response.writeHead(400, {'Content-Type' : 'text/plain' });

response.end('Bad request');

}

function handleRequest(request, response) {

switch (request.method) {

case 'GET':

handleGetRequest(response);

break;

case 'POST':

handlePostRequest(response);

break;

case 'PUT':

handlePutRequest(response);

break;

case 'DELETE':

handleDeleteRequest(response);

break;

default:

handleBadRequest(response);

break;

}

console.log('Request processing completed');

}

http.createServer(handleRequest).listen(8180, '127.0.0.1', () => {

console.log('Started Node.js http server at http://127.0.0.1:8180');

});

When we run this application, our HTTP server will recognize

the GET, POST, PUT, and DELETE HTTP methods, and will handle them in

different functions. To all other HTTP requests, it will gracefully

respond with the HTTP 400 BAD REQUEST status code. To interact with the

HTTP applications, we will use Postman, available to download

from https://www.getpostman.com/. It is a lightweight application for

sending HTTP requests to an endpoint, specifying HTTP headers,

and providing payload. Give it a try and execute test requests for

each of the handler functions we implemented previously:

Modularizing code

What we have developed so far is a simple HTTP server application

that listens and processes known request types; however, it is not so

well structured, as the functions handling the requests are not

reusable. Node.js supports modules embracing code isolation and

reusability.

A user-defined module is a logical unit consisting of one or more related

functions. The module can export one or more functions to other components

while keeping other functions visible only to itself.

We will rework our HTTP server application in such a way that the

entire request handling functionality will be wrapped in a module.

The module will export only a generic handler function that will

take a request object as argument and, based on its request type, it

will delegate the handling to inner functions not visible outside the

module.

Let's start by creating a new module directory within our project.

We will refactor our previous source file by extracting the following

functions to a new http-module.js file inside the newly created

directory:

function handleGetRequest(response) {

response.writeHead(200, {'Content-Type' : 'text/plain'});

response.end('Get action was requested');

}

function handlePostRequest(response) {

response.writeHead(200, {'Content-Type' : 'text/plain'});

response.end('Post action was requested');

}

function handlePutRequest(response) {

response.writeHead(200, {'Content-Type' : 'text/plain'});

response.end('Put action was requested');

}

function handleDeleteRequest(response) {

response.writeHead(200, {'Content-Type' : 'text/plain'});

response.end('Delete action was requested');

}

function handleBadRequest(response) {

console.log('Unsupported http mehtod');

response.writeHead(400, {'Content-Type' : 'text/plain' });

response.end('Bad request');

}

exports.handleRequest = function(request, response) {

switch (request.method) {

case 'GET':

handleGetRequest(response);

break;

case 'POST':

handlePostRequest(response);

break;

case 'PUT':

handlePutRequest(response);

break;

case 'DELETE':

handleDeleteRequest(response);

break;

default:

handleBadRequest(response);

break;

}

console.log('Request processing completed');

}

This file creates a user-defined module that exports the handleRequest

function, making it available to the other components. All the other

functions are accessible only within the module. Although the

sample exports only one function, a module can export as many

functions as is feasible.

Let's use the new http-module in the main.js file in the main directory of

our first project. We have to create an http server using the Node.js

built-in http module, and its createServer will pass its handleRequest

function as its argument. It will serve as a callback function that the

server will invoke on each request:

var http = require('http');

var port = 8180;

var httpModule = require('./modules/http-module');

http.createServer(httpModule.handleRequest).listen(8180, '127.0.0.1', () => {

console.log('Started Node.js http server at http://127.0.0.1:8180');

});

We separated the creation of the server socket from the business

logic that handles the incoming requests bound to it. The require

directive is used to import our module. It uses a relative path to it.

Try this version as well by executing another test request with the

Postman tooling.

Luckily, we will not be creating our own HTTP handlers when implementing

our RESTful-enabled applications. The Express framework will do this for

us. The examples in this chapter are meant to provide a clear example of the

Node.js possibilities when it comes to handling HTTP requests and how user

modules are implemented. We will take a detailed look at the Express

framework in Chapter 3, Building a Typical Web API.

Testing Node.js

Now we will extend our project by providing a unit test for the

HTTP module, but before diving into that, let's have a look at how

Node.js supports unit testing in general. At the beginning of this

chapter, we installed the Nodeunit module. Well, it's about time we

started playing around with it.

First, let's create another simple Node.js module that we will use to

implement our first unit test. Then we will move to more advanced

topics, such as mocking JavaScript objects and using them to create

unit tests for our HTTP module.

I have chosen to develop a simple math module that exports

functions for adding and subtracting integer numbers, as it is

straightforward enough and the results of each operation are

strictly defined.

Let's start with the module and create the following math.js file in our

module directory:

exports.add = function (x, y) {

return x + y;

};

exports.subtract = function (x, y) {

return x - y;

};

The next step will be to create a test-math.js file in the test

subdirectory of our project:

var math = require('../modules/math');

exports.addTest = function (test) {

test.equal(math.add(1, 1), 2);

test.done();

};

exports.subtractTest = function (test) {

test.equals(math.subtract(4,2), 2);

test.done();

};

Finally, run the test module from a shell Terminal with nodeunit

test/test-math.js. The output will show the results of all the test

methods, specifying whether they passed successfully or not:

nodeunit test/test-math.js

test-math.js

addTest

subtractTest

OK: 2 assertions (5ms)

Let's modify addTest in such a way that it gets broken and to see how

test failures are reported by the Nodeunit module:

exports.test_add = function (test) {

test.equal(math.add(1, 1), 3);

test.done();

};

Executing the test this time results in a failure with some assert

failure messages, and in the end, there is an aggregation saying how

many of the executed tests failed:

nodeunit test-math.js

test-math.js

addTest

at Object.equal (/usr/lib/node_modules/nodeunit/lib/types.js:83:39)

at Object.exports.addTest (../hello-node/test/test-math.js:

(..)

AssertionError: 2 == 3

subtractTest

FAILURES: 1/2 assertions failed (12ms)

We just created our Nodeunit's first unit test. However, it tests

math function in a rather isolated way. I suppose you are wondering

how we can use Nodeunit to test functions with complex arguments

such as HTTP request and response, that are bound to a context.

This is possible using so-called mock objects. They are a

predefined version of the state of complex context-based arguments

or functions, in objects that we want to use in our unit test in order

to test the behavior of our module for the exact state of the object.

To use mock objects, we will need to install a module that supports

object mocking. There are various types of testing tools and

modules available out there. Most of them, however, are designed

to test the JavaScript client functionality. There are modules such

as JsMockito, a JavaScript fork of the famous Mockito framework

for Java, and node-inspector, a module that provides a JavaScript

debugger that starts implicitly in the Google Chrome browser.

Native support for the Chrome browser is logical, since Node.js is

built on top of the Google V8 JavaScript Engine. As we are

developing a server-side application, these are not the most

convenient tools, as JsMockito is not pluggable as a Node.js

module, and using a debugger within your browser to debug backed

applications just doesn't seem right to me. Anyway, if you are about

to dive deeper into Node.js, you should definitely give them a try.

For testing server-side JavaScript modules, we will use the Sinon.JS module.

Like all the other modules, it is available in the npm repository, so execute

the following command to install it:

npm install -g sinon

Sinon.JS is a very flexible JavaScript testing library providing

functionality for mocking, stubbing, and spying on JavaScript

objects. It is available at http://sinonjs.org and can be used with any

JavaScript testing framework. Let's see what we need in order to

test our HTTP module. It exports a single method, handleRequest,

which takes the HTTP request and response objects as arguments.

Based on the requested method, the module calls its internal

functions to handle different requests. Each request handler writes

a different output to the response.

To test this functionality in an isolated environment such as

Nodeunit, we need mock objects, which will then be passed as

arguments. To ensure that the module behaves as expected, we will

need to have access to the data stored in those objects.

Working with mock objects

Here are the steps that need to be carried out when using mock

objects:

1. Call the require function with sinon as a parameter and export

a test function from it:

var sinon = require('sinon');

exports.testAPI(test){...}

2. Define an API description of the method you want to mock

as follows:

var api = {'methodX' : function () {},

'methodY' : function() {},

'methodZ' : function() {}};

3. Use sinon within the exported function in order to create

mock objects out of the api description:

var mock = sinon.mock(api);

4. Set the expectations on the mock objects. Expectations are

set on the mocked objects by describing how the mocked

method should behave, what arguments it is supposed to

take, and what value it is supposed to return. When the

mocked method is called with a different state than what is

described, the expectation will fail when verified later:

mock.expects('methodX').once().withArgs('xyz')

.returns('abc');

api.methodX('xyz')

5. The preceding sample expects that methodX gets called exactly

once with the xyz argument, and it will force the method to

return abc. The Sinon.JS module makes that possible for us.

The method of the description object is called and not that of the mocked

object. The mocked object is used to set the expectations for the mocked

method, and later to check whether those expectations have been fulfilled.

6. Use the mocked object in the test environment and, later,

call its verify() method. This method will check whether the

code being tested interacted correctly with mock, that is,

how many times the method has been called and whether it

has been called with the expected arguments. If any of the

expectations is not met, then an error will be thrown,

causing the test to fail.

7. The exported test function of our test module has an

argument. That argument provides assert methods that can

be used to check test conditions. In our example, we mocked

the method to always return abc when called with the 'xyz'

arguments. So, to complete the test, the following assert can

be done, and in the end, the mock object needs to be

verified:

mock.expects('methodX').once().withArgs('xyz')

.returns('abc');

test.equals(api.methodX('xyz'), 'abc');

mock.verify();

8. Try modifying the arguments passed to methodX such that they

don't match the expectation, and you will see this breaking

your test.

9. Let's put these steps into practice and create the following

test-http-module.js file in the test directory:

var sinon = require('sinon');

exports.handleGetRequestTest = (test) => {

var response = {'writeHead' : () => {}, 'end': () => {}};

var responseMock = sinon.mock(response);

responseMock.expects('end').once().withArgs('Get action was

requested');

responseMock.expects('writeHead').once().withArgs(200, {

'Content-Type' : 'text/plain'});

var request = {};

var requestMock = sinon.mock(request);

requestMock.method = 'GET';

var http_module = require('../modules/http-module');

http_module.handleRequest(requestMock, response);

responseMock.verify();

test.done();

};

10. Start the test with Nodeunit's test-http-module.js to verify that it

passes successfully. Your next step will be to extend the test

so that it covers the handling of all the HTTP methods in our

HTTP module:

nodeunit test/test-http-module.js

test-http-module.js

Request processing completed

handleGetRequestTest

OK: 0 assertions (32ms)

Deploying an application

Node.js has an event-driven, non-blocking I/O model, which makes

it perfect for real-time applications that scale well in distributed

environments, such as public or private cloud platforms. Each cloud

platform offers tools that allow seamless deployment, distribution,

and scaling of its hosted applications. In this section, we will look at

two publicly available Node.js application cloud providers—

Nodejitsu and Microsoft Azure.

But first, let's spend some time on the clustering support, as it is

fundamental for understanding why Node.js fits so well into the

cloud environment. Node.js comes with clustering support built in

to its core. Using the cluster module in your applications allows

them to start as many workers as necessary to handle the load they

will face. Generally, it is recommended to match the number of

workers to the number of threads or logical cores your environment

has.

The heart of your application is the master process. It is responsible for

keeping a registry of active workers and the load of the application, and how

to create it. It also creates more workers when needed and reduces them

when the load decreases.

The cloud platform should also ensure that there is zero downtime

when deploying new versions of the applications. In such cases, the

master process needs to be notified that a newer version should be

distributed. It should fork the new workers' new application

version, and notify the workers currently running with the old

version to close their listeners; thus, it stops accepting connections

and exits gracefully once they finish. Thus, all the new incoming

requests will be handled by the newly-started workers, and after the

obsolete workers terminate, all the running workers will be running

the latest version.

Nodejitsu

Let's take a closer look at some of the Node.js Platform as a

Service (PaaS) offerings. The first PaaS we will look at is

Nodejitsu, available at https://www.nodejitsu.com.

This allows seamless deployment of Node.js applications on the

cloud, with many useful features for development, management,

deployment, and monitoring of Node.js applications. To interact

with jitsu, you need to install its command-line interface, which is

available as a Node.js module:

npm install -g jitsu

After installing jitsu and starting it with jitsu, you will be given a

warm welcome, with a friendly console screen that will introduce

you to the basic jitsu commands, as follows:

In order to interact with jitsu, you will need to sign up for it. Jitsu

offers different pricing plans, as well as free trial services.

You can do that either from their website or with the jitsu signup

command. Then you can start making use of the tools the

command-line interface offers.

Microsoft Azure

Microsoft's cloud platform as a service, Azure, also offers hosting of

Node.js applications. They have chosen a slightly different

approach, and instead of providing a command-line interface to

interact with their repositories, they make use of their Git

integration; that is, you interact with Azure as you would interact

with any other Git repository. If you are not familiar with Git, I

strongly recommend that you learn more about this distributed

source code version control system.

If you've chosen Azure as your platform, you will find the following link very

useful: http://azure.microsoft.com/en-us/develop/nodejs/.

Heroku

Heroku is a public cloud offering that allows you to manage, deploy,

and scale Node.js applications. Preparing your Node application for

the Heroku environment does not take too much effort, as long as

you install its command-line interface, available either at https://devce

nter.heroku.com/articles/heroku-cli or through your package manager with

the following:

npm install -g heroku-cli

All you have to do is provide a 'start script' element in the package.json

file, push it to your relevant origin Git repository using git push master

heroku, then log in and create your application, using the heroku login

and heroku create commands.

Self-test questions

To get additional confidence about your newly-gained knowledge,

go through the next set of statements and state whether they are

true or false:

1. Node modules can export more than one function to outer

components

2. Node modules are extensible

3. Modules always need to explicitly declare their

dependencies to other modules

4. When using mocking in a test environment, the mocked

method is called on the mocked object

5. Debugging Node.js code is not as straightforward as other

pieces of non-JavaScript code

Summary

In this chapter, you gained your first Node.js experience, starting

from a simple Hello world application and moving on to a more

complex sample HTTP server-like application that handles

incoming HTTP requests. Being more confident with Node.js, you

refactored the application to use user modules, and then created

unit tests for your module using a mocking framework to eliminate

dependencies on complex objects in your test environment.

Now that you've understood how to handle and test incoming HTTP

requests, in the next chapter, our next step will be to define what a

typical web API looks like and how it can be tested.

Building a Typical Web API

Our first draft API will be a read-only version and will not support

creating or updating items in the catalog as real-world applications

do. Instead, we will concentrate on the API definition itself, and will

worry about data storage later on. Of course, using file storage for

data exposed to millions of users is anything but an option, so a

database layer will be provided to our application further in the

book, after we have looked into modern NoSQL database solutions.

We will also cover the topic of content negotiation, a mechanism

that allows consumers to specify the expected format of requested

data. Finally, we will take a look at several ways to expose different

versions of a service, in case it evolves in a backward-incompatible

way.

To sum up, in this chapter, you will learn the following:

How to specify a web API

How to implement routes

How to query your API

Content negotiation

API versioning

After this chapter, you should be able to completely specify a

RESTful API and will be almost ready to start implementing real-

life Node.js RESTful services.

Specifying the API

The very first thing a project usually starts with is a definition of the

operations the API will expose. According to the REST principles,

an operation is exposed by an HTTP method and a URI. The action

performed by each operation should not contradict the natural

meaning of its HTTP method. The following table specifies the

operations of our API in detail:

tho

URI Description

GET /category

Retrieves all available categories in the

catalog.

GET

/category/{categor

y-id}/

Retrieves all the items available under a

specific category.

GET

/category/{catego

ry-id}/{item-id}

Retrieves an item by its ID under a

specific category.

POST /category

Creates a new category; if it exists, it will

update it.

POST

/category/{categor

y-id}/

Creates a new item in a specified category.

If the item exists, it will update it.

PUT

/category/{catego

ry-id}

Updates a category.

PUT

/category/{categor

y-id}/{item-id}

Updates an item in a specified category.

DELET

/category/{categor

y-id}

Deletes an existing category.

DELET

/category/{categor

y-id}/{item-id}

Deletes an item in a specified category.

The second step is to choose an appropriate format for our catalog

application's data. JSON objects are natively supported by

JavaScript. They are easy to extend during the evolution of an

application and are consumable by almost any platform available.

Thus, the JSON format seems to be our logical choice for us. Here is

the JSON representation of an item, and category objects that will

be used throughout this book:

{

"itemId": "item-identifier-1",

"itemName": "Sports Watch",

"category": "Watches",

"categoryId": 1,

"price": 150,

"currency": "EUR"

}

{

"categoryName" : "Watches",

"categoryId" : "1",

"itemsCount" : 100,

"items" : [{

"itemId" : "item-identifier-1",

"itemName":"Sports Watch",

"price": 150,

"currency" : "EUR"

}]

}

So far, our API has defined a set of operations and the data format

to be used. The next step is to implement a module that will export

functions serving each of the operations in the route.

To begin with, let's create a new Node.js Express project. Select a

directory where your projects will be stored and from your shell

Terminal, execute express chapter3. If you are using Windows, you will

need to install the express-generator module before generating the

project. The express-generator will create your an initial express project

layout in the selected directory. This layout provides the default

project structure for you, ensuring that your Express project follows

the standard project structure. It makes your project easier to

navigate.

The next step is to import the project into the Atom IDE. Right-click

anywhere in the Projects tab and select Add project folder then

select the directory Express generated for you.

As you can see, Express has done some background work for us and

has created a starting point for our application: app.js. It has also

created the package.json file for us. Let's take a look at each of these

files, starting with package.json:

{

"name": "chapter3",

"version": "1.0.0",

"description": "",

"main": "app.js",

"scripts": {

"test": "test"

"author": "",

"license": "ISC",

"dependencies": {

"body-parser": "~1.13.2",

"cookie-parser": "~1.3.5",

"debug": "~2.2.0",

"express": "~4.16.1",

"jade": "~1.11.0",

"morgan": "~1.6.1",

"serve-favicon": "~2.3.0"

}

As we created a blank Node.js Express project, we initially have

dependencies only to the Express framework, some middleware

modules such as morgan, body-parser, and cookie-parser, and the Jade

template language. Jade is a straightforward template language

used to produce HTML code inside templates. If you are interested

in it, you can find out more about it at http://www.jade-lang.com.

The current version of the Express framework at the time of writing

is 4.16.1; to update it, execute npm install express@4.16.1 --save from the

chapter3 directory. This command will update the dependency of the

application to the desired version. The --save option will update and

save the new version of the dependency in the project's package.json

file.

When you introduce new module dependencies, it is up to you to keep the

package.json file up to date in order to maintain an accurate state of the

modules your application depends on.

We will come to what middleware modules are a bit later in the

chapter.

For now, we will ignore the content of the public and view directories

as it is not relevant to our RESTful service. They contain the auto-

generated stylesheets and template files that might be helpful, if we

decide to develop a web-based consumer of the services at a later

stage.

We've already mentioned that the Express project created a starting

point for our web application in app.js. Let's take a deeper look at it:

var express = require('express');

var path = require('path');

var favicon = require('serve-favicon');

var logger = require('morgan');

var cookieParser = require('cookie-parser');

var bodyParser = require('body-parser');

var routes = require('./routes/index');

var users = require('./routes/users');

var app = express();

// view engine setup

app.set('views', path.join(__dirname, 'views'));

app.set('view engine', 'jade');

// uncomment after placing your favicon in /public

//app.use(favicon(path.join(__dirname, 'public', 'favicon.ico')));

app.use(logger('dev'));

app.use(bodyParser.json());

app.use(bodyParser.urlencoded({ extended: false }));

app.use(cookieParser());

app.use(express.static(path.join(__dirname, 'public')));

app.use('/', routes);

app.use('/users', users);

// catch 404 and forward to error handler

app.use(function(req, res, next) {

var err = new Error('Not Found');

err.status = 404;

next(err);

});

// error handlers

// development error handler

// will print stacktrace

if (app.get('env') === 'development') {

app.use(function(err, req, res, next) {

res.status(err.status || 500);

res.render('error', {

message: err.message,

error: err

});

}

// production error handler

// no stacktraces leaked to user

app.use(function(err, req, res, next) {

res.status(err.status || 500);

res.render('error', {

message: err.message,

error: {}

});

module.exports = app;

Obviously, the Express generator has done a lot for us as it has

instantiated the Express framework and has assigned a complete

development environment around it. It has done the following:

Configured the middleware to be used in our

application, body-parser, the default router, as well as error

handler middleware for our development environment

Injected a logger instance of the morgan middleware module

Configured the Jade template, as it has been selected as the

default template for our application

Configured the default URI that our Express application will

be listening to, / and /users, and created dummy handle

functions for them

You will have to install all the modules used in app.js in order to start

the generated application successfully. Also, make sure you update

the dependencies of your package.json file using the --save option after

installing them.

The Express generator also created a starting script for the

application. It is under the bin/www directory of your project and

looks like the following snippet:

#!/usr/bin/env node

/**

* Module dependencies.

var app = require('../app');

var debug = require('debug')('chapter3:server');

var http = require('http');

/**

* Get port from environment and store in Express.

var port = normalizePort(process.env.PORT || '3000');

app.set('port', port);

/**

* Create HTTP server.

var server = http.createServer(app);

/**

* Listen on provided port, on all network interfaces.

server.listen(port);

server.on('error', onError);

server.on('listening', onListening);

/**

* Normalize a port into a number, string, or false.

function normalizePort(val) {

var port = parseInt(val, 10);

if (isNaN(port)) {

// named pipe

return val;

}

if (port >= 0) {

// port number

return port;

}

return false;

}

/**

* Event listener for HTTP server "error" event.

function onError(error) {

if (error.syscall !== 'listen') {

throw error;

}

var bind = typeof port === 'string'

? 'Pipe ' + port

: 'Port ' + port;

// handle specific listen errors with friendly messages

switch (error.code) {

case 'EACCES':

console.error(bind + ' requires elevated privileges');

process.exit(1);

break;

case 'EADDRINUSE':

console.error(bind + ' is already in use');

process.exit(1);

break;

default:

throw error;

}

/**

* Event listener for HTTP server "listening" event.

function onListening() {

var addr = server.address();

var bind = typeof addr === 'string'

? 'pipe ' + addr

: 'port ' + addr.port;

debug('Listening on ' + bind);

}

To start the application, execute node bin/www; this will execute the

script above and will start the Node.js application. So requesting

http://localhost:3000 in your browser will result in calling the

default GET handler, which gives a warm welcome response:

Default welcome message from an Express application

The generator created a dummy routes/users.js; it exposes a route

linked to a dummy module available at the /users location.

Requesting it will result in calling the list function of the user's

route, which outputs a static response: respond with a resource.

Our application will not be using a template language and style

sheets, so let's get rid of the lines that set the views and view engine

properties in the application configuration. In addition, we will be

implementing our own routes. Thus, we don't need the binding of

the / and /users URIs for our app, neither do we need the user

module; instead, we will utilize a catalog module and from a route:

var express = require('express');

var path = require('path');

var favicon = require('serve-favicon');

var logger = require('morgan');

var cookieParser = require('cookie-parser');

var bodyParser = require('body-parser');

var routes = require('./routes/index');

var catalog = require('./routes/catalog')

var app = express();

//uncomment after placing your favicon in /public

//app.use(favicon(path.join(__dirname, 'public', 'favicon.ico')));

app.use(logger('dev'));

app.use(bodyParser.json());

app.use(bodyParser.urlencoded({ extended: false }));

app.use(cookieParser());

app.use(express.static(path.join(__dirname, 'public')));

app.use('/', routes);

app.use('/catalog', catalog);

// catch 404 and forward to error handler

app.use(function(req, res, next) {

var err = new Error('Not Found');

err.status = 404;

next(err);

});

//development error handler will print stacktrace

if (app.get('env') === 'development') {

app.use(function(err, req, res, next) {

res.status(err.status || 500);

res.render('error', {

message: err.message,

error: err

});

}

// production error handler no stacktraces leaked to user

app.use(function(err, req, res, next) {

res.status(err.status || 500);

res.render('error', {

message: err.message,

error: {}

});

module.exports = app;

So after this cleanup, our application looks a lot cleaner and we are

ready to move forward.

Before doing that, though, there is one term that needs further explanation:

middleware. It is a subset of chained functions called by the Еxpress.js routing

layer before а user-defined handler is invoked. Middleware functions have

full access to the request and response objects and can modify either of them. The

middleware chain is always called in the exact order in which it has been

defined, so it is vital for you to know exactly what a specific piece of

middleware is doing. Once a middleware function finishes, it calls the next

function in the chain by invoking its next argument as a function. After the

complete chain gets executed, the user-defined request handler is called.

Here are the basic rules that apply to the middleware chain:

A middleware function has the following signature: function

(request, response, next).

Middleware functions are executed in the exact order in

which they have been added to the application chain. This

means that if you want your middleware function to be

called before a specific route, you need to add it before

declaring the route.

Middleware functions use their third parameter, next, as a

function to indicate that they have completed their work and

to exit. When the next() parameter of the last function in the

chain has been called, the chained execution is completed

and the request and the response objects reach the defined

handlers, in the state set by the middleware.

Now that we know what a middleware function is, let's clarify what

the currently used middleware functions provide our application

with. The body-parser middleware is the Express framework built in a

parser. It parses the request body and populates the request object

after the middleware execution finishes, that is, it provides JSON

payload handling.

Now it is time to move on and implement our user module that will

be mapped to our URIs. The module will be named modules/catalog.js:

var fs = require('fs');

function readCatalogSync() {

var file = './data/catalog.json';

if (fs.existsSync(file)) {

var content = fs.readFileSync(file);

var catalog = JSON.parse(content);

return catalog;

}

return undefined;

}

exports.findItems = function(categoryId) {

console.log('Returning all items for categoryId: ' + categoryId);

var catalog = readCatalogSync();

if (catalog) {

var items = [];

for (var index in catalog.catalog) {

if (catalog.catalog[index].categoryId === categoryId) {

var category = catalog.catalog[index];

for (var itemIndex in category.items) {

items.push(category.items[itemIndex]);

}

return items;

}

return undefined;

}

exports.findItem = function(categoryId, itemId) {

console.log('Looking for item with id' + itemId);

var catalog = readCatalogSync();

if (catalog) {

for (var index in catalog.catalog) {

if (catalog.catalog[index].categoryId === categoryId) {

var category = catalog.catalog[index];

for (var itemIndex in category.items) {

if (category.items[itemIndex].itemId === itemId) {

return category.items[itemIndex];

}

return undefined;

}

exports.findCategoryies = function() {

console.log('Returning all categories');

var catalog = readCatalogSync();

if (catalog) {

var categories = [];

for (var index in catalog.catalog) {

var category = {};

category["categoryId"] = catalog.catalog[index].categoryId;

category["categoryName"] = catalog.catalog[index].categoryName;

categories.push(category);

}

return categories;

}

return [];

}

The catalog module is built around the catalog.json file, stored in the

data directory. The content of the source file is read synchronously

using the File System module, fs, within the readCatalogSync function.

The File System module provides multiple useful filesystem

operations such as functions for creating, renaming, or deleting files

or directories; truncating; linking; chmod functions; as well as

synchronous and asynchronous file access for reading and writing

data. In our sample application, we aim to use the most

straightforward approach, so we implement functions that read the

catalog.json file by utilizing the readFileSync function of the File System

module. It returns the content of a file as a string, within a

synchronous call. All other functions of the module are exported

and can be used to query the content of the source file, based on

different criteria.

The catalog module exports the following functions:

findCategories: This returns an array of JSON objects

containing all the categories in the catalog.json file

findItems (categoryId): This returns an array JSON objects

representing all the items in a given category

findItem(categoryId, itemId): This returns a JSON object

representing a single item in a given category

Now that we have three complete functions, let's see how to bind

them to our Express application.

Implementing routes

In Node.js terms, a route is a binding between a URI and function.

The Express framework provides built-in support for routing. An

express object instance contains functions named after each HTTP

verb: get, post, put, and delete. They have the following syntax:

function(uri, handler);. They are used to bind a handler function to a

specific HTTP action executed over a URI. The handler function

usually takes two arguments: request and response. Let's see it with a

simple Hello route application:

var express = require('express');

var app = express();

app.get('/hello', function(request, response){

response.send('Hello route');

});

app.listen(3000);

Running this sample at localhost and accessing

http://localhost:3000/hello will result in calling your handler function

and it will respond saying Hello route, but routing can give you much

more. It allows you to define a URI with parameters; for example,

let's use /hello/:name as a routing string. It tells the framework that the

URI used consists of two parts: a static part (hello) and a variable

part (the name parameter).

Furthermore, when the routing string and the handler function are

defined in line with the get function of an Express instance, a

parameter collection is made available directly in the request

argument of the handler function. To demonstrate this, let's modify

our previous example a bit:

var express = require('express');

var app = express();

app.get('/hello:name', function(request, response){

response.send('Hello ' + request.params.name);

});

app.listen(3000);

As you can see in the preceding code snippet, we used a colon (:) to

separate the parameter part of the URI from the static part. You can

have multiple parameters in an Express route; for

example, /category/:category-id/items/:item-id defines a route for

displaying an item that belongs to a category, where the category-id

and item-id are parameters.

Now let's try it out. Requesting http://localhost:3000/hello/friend will

result in the following output:

hello friend

This is how we can provide parameterized URIs with Express. It is a

nice feature, but it is often not enough. In web applications, we are

used to providing additional parameters with GET parameters.

Unfortunately, the Express framework is not so good with GET

parameters. Thus, we have to utilize the url module. It is built into

Node.js to provide an easy way of using URL parsing. Let's use our

hello result with other parameters in the application again, but

extend it in a way that it outputs hello all when /hello is requested

and hello friend when the requested URI is /hello?name=friend:

var express = require('express');

var url = require('url');

var app = express();

app.get('/hello', function(request, response){

var getParams = url.parse(request.url, true).query;

if (Object.keys(getParams).length == 0) {

response.end('Hello all');

} else {

response.end('Hello ' + getParams.name);

}

});

app.listen(3000);

There are a few things worth mentioning here. We used the url

module's function parse. It takes a URL as its first argument and a

Boolean as an optional second argument, which specifies whether

the query string should be parsed or not. The url.parse function

returns an associative object. We used Object.keys with it to transform

the keys in these associative objects into an array so that we can

check its length. This will help us check whether our URI has been

called with GET parameters or not. In addition to the routing

functions named after each HTTP verb, there is also a function

named all. When used, it routes all the HTTP actions to the

specified URI.

Now that we know how routing and the GET parameters work within

Node.js and the Express environment, we are ready to define a

route for the catalog module and bind it in our application. The

following is the route as defined in routes/catalog.js.

var express = require('express');

var catalog = require('../modules/catalog.js')

var router = express.Router();

router.get('/', function(request, response, next) {

var categories = catalog.findCategoryies();

response.json(categories);

});

router.get('/:categoryId', function(request, response, next) {

var categories = catalog.findItems(request.params.categoryId);

if (categories === undefined) {

response.writeHead(404, {'Content-Type' : 'text/plain'});

response.end('Not found');

} else {

response.json(categories);

}

});

router.get('/:categoryId/:itemId', function(request, response, next) {

var item = catalog.findItem(request.params.categoryId,

request.params.itemId);

if (item === undefined) {

response.writeHead(404, {'Content-Type' : 'text/plain'});

response.end('Not found');

} else {

response.json(item);

}

});

module.exports = router;

First, a Router instance is created from the Express module. Here is a

table that nicely describes the routing we just implemented. This

will be helpful later when we test our API:

HTTP

method

Route

Catalog's module

function

GET /catalog findCategories()

GET /catalog/:categoryId findItems(categoryId)

GET

/catalog/:categoryId/:it

emId

findItem(categoryId, itemId)

Querying the API using test

data

We need some test data in order to test our service, so let's use the

catalog.json file in the data directory of our project. This data will allow

us to test all our three functions, but to do that, we would need a

client that can send REST requests against an endpoint. If you still

haven't created a Postman project for testing your application, now

is an appropriate time to create it.

Requesting /catalog should return all the categories in the test file:

Thus, requesting /catalog/1 should result in returning a list with all

the items under the Watches category:

Finally, requesting http://localhost:3000/catalog/1/item-identifier-1 would

display only the item identified by item-identifier-1, and requesting a

nonexistent item would result in response with status code 404:

Content negotiation

So far, the catalog service supports only the JSON format, and thus

works only with the media type application/json. Let's assume our

service has to offer data in different formats, for example, both

JSON and XML. Then, the consumer needs to explicitly define the

data format they need. The best way to carry out content

negotiation in REST has been a very debatable subject for a long

time.

In his famous discussion on implementing content negotiation

correctly, Roy Fielding states the following:

All important resources must have URIs.

However, that leaves a gap on how to expose the same resource in a

different data format, so Roy continues with the following:

Agent-driven negotiation is far more effective, but there was a huge disagreement

between myself and the chair of the HTTP working group and my original agent-driven

design for HTTP/1.1 was effectively buried in committee. To do negotiation right, the

client needs to be aware of all the alternatives and what it should use as a bookmark.

While one can still choose to stick with a URI-driven negotiation by

providing the desired format with custom GET parameters, the REST

community has chosen to stick to Roy's suggestion for agent-driven

negotiation. Now that it has been almost a decade since this

argument was initiated, it has been proven that they took the right

decision. Agent-driven negotiation makes use of the Accept HTTP

header.

The Accept HTTP header specifies the media type of the resource that

the consumer is willing to process. In addition to the Accept header,

the consumer may also make use of the Accept-Language and Accept-

Encoding headers to specify what language and encoding the results

should be provided in. If the server fails to provide the results in the

expected format, it can either return a default value or make use of

HTTP 406 Not acceptable in order not to cause data confusion errors on

the client side.

The Node.js HTTP response object contains a method, format, that

performs content negotiation based on the Accept HTTP header if set

in the request object. It uses the built-in request.accepts() to select an

appropriate handler for the request. If that is not found, the server

invokes the default handler, which responds with HTTP 406 Not

acceptable. Let's create a demo on how to use the format method within

one of our routes. For that purpose, let's assume we have

implemented a function within our catalog module, named

list_groups_in_xml, that provides the group data in XML format:

app.get('/catalog', function(request, response) {

response.format( {

'text/xml' : function() {

response.send(catalog.findCategoiesXml());

'application/json' : function() {

response.json(catalog.findCategoriesJson());

'default' : function() {.

response.status(406).send('Not Acceptable');

}

});

This is how you can implement content negotiation in a clear and

straightforward way.

API versioning

It is an inevitable fact that all application APIs evolve. However, the

evolution of public APIs with an unknown number of consumers,

such as RESTful services, is a sensitive topic. As consumers may not

be able to handle the modified data appropriately and there is no

way of notifying all of them, we need to keep our APIs as backward-

compatible as possible. One way to do so is to use different URIs for

different versions of our application. Currently, our catalog API is

available at /catalog.

When the time is right for a new version, for example, Version 2, we

may need to keep the previous version available at another URI for

backward-compatibility. Best practice is to encode the version

number in the URI, such as /v1/catalog, and keep /catalog mapped to

the latest version. Thus, requesting /catalog will cause a redirect to

/v2/catalog and will make use of the HTTP 3xx status codes to indicate

the redirection to the latest version.

Another option for versioning would be to keep the URI of your API

stable and rely on custom HTTP headers that will specify the

version. But that's not very stable approach concerning backward-

compatibility as it is much more natural to modify a URL of a

request in an application rather than modify the headers that are

sent within the request.

Self-test questions

To get additional confidence, go through this set of statements and

state whether they are true or false:

1. A REST-enabled endpoint must support all HTTP methods

relevant to the REST principles

2. When content negotiation fails, due to an unsupported

media type passed as a value of the accepts header, 301 is

the appropriate status code.

3. When using parameterized routes, the developer can specify

the type of the parameter, for example, whether it is a

numeric or a literal type.

Summary

In this chapter, we dived into some more complex topics. Let's sum

up what we covered. We started by specifying the operations of our

web API and defined that an operation is a combination of a URI

and HTTP action. Next, we implemented routes and bound them to

an operation. Then, we requested each operation using the Postman

REST client to request the URIs that we routed. In the content

negotiation section, we handled the Accept HTTP header to provide

the results in the format requested by consumers. Finally, we

covered the topic of API versions, which allow us to develop

backward-compatible APIs.

We used old-fashioned filesystem storage for our data in this

chapter. This is not suitable for a web application. Thus, we will

look into modern, scalable, and reliable NoSQL storage in the next

chapter.

Using NoSQL Databases

In the last chapter, we implemented a sample application exposing

a read-only service that provided catalog data. For the sake of

simplicity, we introduced a performance bottleneck in this

implementation by using file storage. This storage is not suitable for

web applications. It relies on 33 physical files, preventing our

application from servicing heavy loads, as file storage lacks

multitenant support due to disc I/O operations. In other words, we

definitely need to look for a better storage solution, which scales

easily, when needed, following the demands of our REST-enabled

application. NoSQL databases are now heavily used in web and in

cloud environments, ensuring zero downtime and high availability.

They have the following advantages over traditional transactional

SQL databases:

They support schema versions; that is, they can work

with object representations rather than filling the object

state based on definitions of one or several tables.

They are extensible, because they store an actual object.

Data evolution is supported implicitly, so all you need to do

is call the operation that stores the modified object.

They are designed to be highly distributed and scalable.

Nearly all modern NoSQL solutions out there support clustering

and can scale further, along with the load of your application.

Additionally, most of them have REST-enabled interfaces over

HTTP, which eases their usage over a load balancer in high-

availability scenarios. Classical database drivers are usually not

available for traditional client-side languages, such as JavaScript,

because they require native libraries or drivers. However, the idea

of NoSQL originated from using document data stores. Thus, most

of them support the JSON format, which is native to JavaScript.

Last but not least, most NoSQL solutions are open source and are

available for free, with all the benefits that open source projects

offer: community, examples, and freedom!

In this chapter, we will take a look at the MongoDB NoSQL

database and the Mongoose module that interacts with it. We will

see how to design and implement automated tests for the database

models. Finally, at the end of the chapter, we will remove the file

storage bottleneck and will move our application to an almost ready

for production state.

MongoDB – a document store

database

MongoDB is an open source document database with built-in

support for the JSON format. It provides a full index support, based

on any of the available attributes in a document. It is ideal for high-

availability scenarios due to its scalability features. MongoDB,

available at https://mms.mongodb.com, is cloud-ready with its management

services, MongoDB Management Services (MMS). They utilize

and automate most development operations that need to be carried

out to keep your cloud database in good shape, taking care of

upgrades, further scaling, backups, recovery, performance, and

security alerts.

Let's move forward and install MongoDB. Installers for Windows,

Linux, macOS, and Solaris are available at http://www.mongodb.org/download

s. Linux users can find MongoDB in all popular distribution

repositories, while Windows users can make use of a user-friendly

wizard which will guide you through the installation steps, where,

for a typical installation, all you need to do is accept the license

agreement and provide an installation path.

After a successful installation, execute the following command to

start MongoDB. If you want to specify a custom location for your

data, you have to use the --dbpath argument. Optionally, you can start

the MongoDB HTTP console via the --rest argument:

mongod --dbpath ./data --rest

The default port for communicating with MongoDB is 27017, and its

HTTP console is implicitly configured to use a port higher than the

data port by a value of 1,000. Therefore, the default port of the

console will be 28017. The HTTP console provides useful information

about the database, such as logs, health status, available databases,

and so on. I strongly advise you to spend some time with it. The

console can also be used as a RESTful health check service of the

database, because it provides JSON-encoded information about the

running database services and the last error that occurred:

GET /replSetGetStatus?text=1 HTTP/1.1

Host: localhost:28017

Connection: Keep-Alive

User-Agent: RestClient-Tool

HTTP/1.0 200 OK

Content-Length: 56

Connection: close

Content-Type: text/plain;charset=utf-8

{

"ok": 0,

"errmsg": "not running with --replSet"

}

This REST interface can be used in a script or an application to

automate altering notifications providing the current state of the

database engine and so on.

The log section of the console shows that your server is running

successfully (if it is). Now we are ready to move further and see how

to connect Node.js to MongoDB.

Database modeling with

Mongoose

Mongoose is a module that connects Node.js to MongoDB in an

object document mapper (ODM) style. It offers the Create,

Read, Update, and Delete (also known as CRUD)

functionalities for documents stored in the database. Mongoose

defines the structure of the used documents using schemas. The

schema is the smallest unit of data definition in Mongoose. A model

is built out of a schema definition. It is a constructor-like function

that can be used to create or query documents. Documents are

instances of a model and represent one-to-one mapping to the

documents stored in MongoDB. The schema-model-document

hierarchy provides a self-descriptive way of defining objects and

allows easy data validation.

Let's start by installing Mongoose with npm:

npm install mongoose

Now that we have the Mongoose module installed, our first step will

be to define a schema that will represent an item in the catalog:

var mongoose = require('mongoose');

var Schema = mongoose.Schema;

var itemSchema = new Schema ({

"itemId" : {type: String, index: {unique: true}},

"itemName": String,

"price": Number,

"currency" : String,

"categories": [String]

});

The preceding code snippet creates a schema definition of an item.

Defining a schema is straightforward and is quite similar to JSON

schema definition; you have to describe and attribute with its type

and optionally provide additional properties for each key. In the

case of the catalog application, we need to use the itemId as a unique

index in order to avoid having two different items with the same ID.

Thus, apart from defining its type as String, we also use the index

attribute to describe that the value of the itemId field must be unique

for each individual item.

Mongoose introduces the term model. A model is a constructor-

like function compiled out of a schema definition. An instance of a

model represents a document that can be saved to or read from the

database. Creating a model instance is done by calling the model

function of a mongoose instance and passing the schema that the model

should use:

var CatalogItem = mongoose.model('Item', itemSchema);

A model also exposes functions for querying and data

manipulations. Assuming that we have initialized a schema and

created a model, storing a new item to MongoDB is as simple as

creating a new model instance and invoking its save function:

var mongoose = require('mongoose');

mongoose.connect('mongodb://localhost/catalog');

var db = mongoose.connection;

db.on('error', console.error.bind(console, 'connection error:'));

db.once('open', function() {

var watch = new CatalogItem({

itemId: 9 ,

itemName: "Sports Watch1",

brand: 'А1',

price: 100,

currency: "EUR",

categories: ["Watches", "Sports Watches"]

});

watch.save((error, item, affectedNo)=> {

if (!error) {

console.log('Item added successfully to the catalog');

} else {

console.log('Cannot add item to the catlog');

}

});

db.once('open', function() {

var filter = {

'itemName' : 'Sports Watch1',

'price': 100

}

CatalogItem.find(filter, (error, result) => {

if (error) {

consoloe.log('Error occured');

} else {

console.log('Results found:'+ result.length);

console.log(result);

}

});

Here is how to use the model in order to query for documents

representing a sports watch belonging to the Watches group named

Sports Watches:

db.once('open', function() {

var filter = {

'itemName' : 'Sports Watch1',

'price': 100

}

CatalogItem.findOne(filter, (error, result) => {

if (error) {

consoloe.log('Error occurred');

} else {

console.log(result);

}

});

The model also exposes a findOne function, a convenient way of

finding an object by its unique index and then performing some

data manipulation on it, that is, for delete or update operations. The

following example deletes an item:

CatalogItem.findOne({itemId: 1 }, (error, data) => {

if (error) {

console.log(error);

return;

} else {

if (!data) {

console.log('not found');

return;

} else {

data.remove(function(error){

if (!error) { data.remove();}

else { console.log(error);}

});

}

});

Testing a Mongoose model

with Mocha

Mocha is one of the most popular testing frameworks for

JavaScript; its main goal is to provide an easy way to test

asynchronous JavaScript code. Let's install Mocha globally so that

we can make it available to any Node.js application that we may

develop in the future:

npm install -g mocha

We will also need an assertion library that can be used together with

Mocha. The assertion library provides functions for validating

actual values, against expected ones, when they are not equal, the

assertion library will cause test failure. Should.js assertion library

module is easy to use and it will be our choice, so let's install it

globally too:

npm install -g should

Now that we have our testing modules installed, we need to specify

our testcase file path in the package.json file. Let's modify it by adding a

test element pointing to Mocha and the testcase file in the script

node:

{

"name": "chapter4",

"version": "0.0.0",

"private": true,

"scripts": {

"start": "node ./bin/www",

"test": "mocha test/model-test.js"

"dependencies": {

"body-parser": "~1.13.2",

"cookie-parser": "~1.3.5",

"debug": "~2.2.0",

"express": "~4.16.0",

"jade": "~1.11.0",

"morgan": "~1.6.1",

"serve-favicon": "~2.3.0"

}

This will tell the npm package manager to trigger Mocha when the

npm test is executed.

Automation of Mongoose tests must not be affected by the current

state of the database. To ensure that the results are predictable at

each test run, we need to ensure that the database state is exactly as

we would expect it. We will implement a module called prepare.js in

the test directory. It will clear the database before each test run:

var mongoose = require('mongoose');

beforeEach(function (done) {

function clearDatabase() {

for (var i in mongoose.connection.collections) {

mongoose.connection.collections[i].remove(function()

{});

}

return done();

}

if (mongoose.connection.readyState === 0) {

mongoose.connect(config.db.test, function (err) {

if (err) {

throw err;

}

return clearDatabase();

});

} else {

return clearDatabase();

}

});

afterEach(function (done) {

mongoose.disconnect();

return done();

});

Next, we will implement a Mocha test, which creates a new item:

var mongoose = require('mongoose');

var should = require('should');

var prepare = require('./prepare');

const model = require('../model/item.js');

const CatalogItem = model.CatalogItem;

mongoose.createConnection('mongodb://localhost/catalog');

describe('CatalogItem: models', function () {

describe('#create()', function () {

it('Should create a new CatalogItem', function (done) {

var item = {

"itemId": "1",

"itemName": "Sports Watch",

"price": 100,

"currency": "EUR",

"categories": [

"Watches",

"Sports Watches"

]

};

CatalogItem.create(item, function (err, createdItem) {

// Check that no error occured

should.not.exist(err);

// Assert that the returned item has is what we expect

createdItem.itemId.should.equal('1');

createdItem.itemName.should.equal('Sports Watch');

createdItem.price.should.equal(100);

createdItem.currency.should.equal('EUR');

createdItem.categories[0].should.equal('Watches');

createdItem.categories[1].should.equal('Sports Watches');

//Notify mocha that the test has completed

done();

});

Executing the npm test now results in a call against the MongoDB

database creating an item out of the passed JSON object. After

insertion, the assert callback will be executing, ensuring that values

passed to by Mongoose are the same as the returned ones from the

database. Give it a try and break the test—simply change the

expected value in the assert with an invalid one—you will see the

test failing.

Creating a user-defined model

around a Mongoose model

After seeing how a model works, it is time to create a user-defined

module that wraps all CRUD operations for the catalog. Since we

intend to use that module in a RESTful web application, it seems

logical to leave the schema definition and the model creation

outside the module and have them provided as arguments of each

module function. The same schema definition is used in the unit

tests, ensuring stability of the module. Now let's add an

implementation for each CRUD function, starting with a remove()

function. It looks up an item based on its id and deletes it from the

database, if it exists:

exports.remove = function (request, response) {

console.log('Deleting item with id: ' + request.body.itemId);

CatalogItem.findOne({itemId: request.params.itemId}, function(error, data)

{

if (error) {

console.log(error);

if (response != null) {

response.writeHead(500, contentTypePlainText);

response.end('Internal server error');

}

return;

} else {

if (!data) {

console.log('Item not found');

if (response != null) {

response.writeHead(404, contentTypePlainText);

response.end('Not Found');

}

return;

} else {

data.remove(function(error){

if (!error) {

data.remove();

response.json({'Status': 'Successfully deleted'});

}

else {

console.log(error);

response.writeHead(500, contentTypePlainText);

response.end('Internal Server Error');

}

});

}

});

}

The saveItem() function takes the request body payload as an

argument. A valid update request will contain the new state of an

item object, represented in JSON format. First, the itemId is parsed

out of the JSON object. Next, lookup is done. If an item exists, it

gets updated. Otherwise, a new one gets created:

exports.saveItem = function(request, response)

{

var item = toItem(request.body);

item.save((error) => {

if (!error) {

item.save();

response.writeHead(201, contentTypeJson);

response.end(JSON.stringify(request.body));

} else {

console.log(error);

CatalogItem.findOne({itemId : item.itemId },

(error, result) => {

console.log('Check if such an item exists');

if (error) {

console.log(error);

response.writeHead(500, contentTypePlainText);

response.end('Internal Server Error');

} else {

if (!result) {

console.log('Item does not exist. Creating a new one');

item.save();

response.writeHead(201, contentTypeJson);

response.

response.end(JSON.stringify(request.body));

} else {

console.log('Updating existing item');

result.itemId = item.itemId;

result.itemName = item.itemName;

result.price = item.price;

result.currency = item.currency;

result.categories = item.categories;

result.save();

response.json(JSON.stringify(result));

}

});

}

});

};

The toItem() function converts the JSON payload to a CatalogItem model

instance, that is, an item document:

function toItem(body) {

return new CatalogItem({

itemId: body.itemId,

itemName: body.itemName,

price: body.price,

currency: body.currency,

categories: body.categories

});

}

We will also need to provide a means of querying data, so let's

implement a function that queries for all items in a category:

exports.findItemsByCategory = function (category, response) {

CatalogItem.find({categories: category}, function(error, result) {

if (error) {

console.error(error);

response.writeHead(500, { 'Content-Type': 'text/plain' });

return;

} else {

if (!result) {

if (response != null) {

response.writeHead(404, contentTypePlainText);

response.end('Not Found');

}

return;

}

if (response != null){

response.setHeader('Content-Type', 'application/json');

response.send(result);

}

console.log(result);

}

});

}

Similar to findItemsByCategory, the following is a function that finds an

item by its ID:

exports.findItemById = function (itemId, response) {

CatalogItem.findOne({itemId: itemId}, function(error, result) {

if (error) {

console.error(error);

response.writeHead(500, contentTypePlainText);

return;

} else {

if (!result) {

if (response != null) {

response.writeHead(404, contentTypePlainText);

response.end('Not Found');

}

return;

}

if (response != null){

response.setHeader('Content-Type', 'application/json');

response.send(result);

}

console.log(result);

}

});

}

Finally, there's a function that lists all the catalog items stored in

the database. It uses the Mongoose model find function that looks

for all documents of the model, and uses its first arguments as a

filter. We want a function that returns all existing documents; that's

why we provide an empty object. This will return all available items.

The results are available in the callback function, which is the second

argument of the model's find function:

exports.findAllItems = function (response) {

CatalogItem.find({}, (error, result) => {

if (error) {

console.error(error);

return null;

}

if (result != null) {

response.json(result);

} else {

response.json({});

}

});

};

The catalog module will be the foundation of our RESTful service. It

is responsible for all data manipulation operations, as well as for

different kinds of queries. It encapsulates all operations in a

reusable way.

Wiring up a NoSQL database

module to Express

Now we have automated tests for the model and a user-defined

module which makes use of them. This ensures the stability of the

module and makes it ready for wider adoption.

It is time to build a new Express-based application and add a route,

exposing the new module to it:

const express = require('express');

const router = express.Router();

const catalog = require('../modules/catalog');

const model = require('../model/item.js');

router.get('/', function(request, response, next) {

catalog.findAllItems(response);

});

router.get('/item/:itemId', function(request, response, next) {

console.log(request.url + ' : querying for ' + request.params.itemId);

catalog.findItemById(request.params.itemId, response);

});

router.get('/:categoryId', function(request, response, next) {

console.log(request.url + ' : querying for ' + request.params.categoryId);

catalog.findItemsByCategory(request.params.categoryId, response);

});

router.post('/', function(request, response, next) {

console.log('Saving item using POST method);

catalog.saveItem(request, response);

});

router.put('/', function(request, response, next) {

console.log('Saving item using PUT method');

catalog.saveItem(request, response);

});

router.delete('/item/:itemId', function(request, response, next) {

console.log('Deleting item with id: request.params.itemId);

catalog.remove(request, response);

});

module.exports = router;

To sum up, we routed each function of the catalog data service

module to an operation of a RESTful service:

GET /catalog/item/:itemId: This calls catalog.findItemById()

POST /catalog: This calls catalog.saveItem()

PUT /catalog: This calls catalog.saveItem()

DELETE / catalog/item/:id: This calls catalog.remove()

GET /catalog/:category: This calls catalog.findItemsByCategory()

GET /catalog/: This calls catalog.findAllItems()

As we have our operations exposed, we are ready to perform some

more serious REST testing. Let's start Postman and test the newly

exposed endpoints:

Spend some time testing each operation thoroughly. This will help

you gain confidence that the catalog data service module really

works, and also will make you more comfortable with how HTTP

responses are served and read. As a RESTful API developer, you

should be able to read HTTP dumps, which show different request

payloads and status codes, fluently.

Self-test questions

Go through the following questions:

How would you perform a query with Mongoose for a single

value of a multivalued attribute?

Define a strategy for testing a Node.js module manipulating

a NoSQL database.

Summary

In this chapter, we looked at MongoDB, a powerful document-

oriented database. We utilized it and made use of Mocha to

implement an automated test for the database layer. Now it is time

to build a fully-fledged RESTful web service. In the next chapter, we

will extend the user-defined module by including support for

searching via document attributes, and add filtering and pagination

capabilities that will finally evolve into full-fledged RESTful service

implementation.

Restful API Design Guidelines

In the previous chapter, we implemented a catalog module that

exposed functions for data manipulation for the items in the catalog

application. These functions made use of the express.js request

object to parse the incoming data in the body and then to carry out

the appropriate database operation. Each function populated

the response object with a relevant status code and response body

payload, if needed. Finally, we bound each function to a route,

accepting HTTP requests.

Now, it's time to look closer into the URLs of the routes and into the

returned HTTP status codes per operation.

In this chapter, we will cover the following topics:

Endpoint URLs and HTTP status codes best practices

Extensibility and versioning

Linked data

Endpoint URLs and HTTP

status codes best practices

Each RESTful API operation is a combination of an HTTP request

against a URL and an appropriate HTTP method.

When executed, each operation will return a status code, indicating

whether the invocation has been successful or not. Successful

invocation is indicated by a HTTP 2XX status code, while

operations that are not executed correctly indicate this

with erroneous status code—4XX if the error is at client side, or 5xx

when the server fails to process a valid request.

Having a well-specified API is vital for its adoption. Such

specification should not only completely enumerate the status codes

of each operation, but should also specify the expected data format,

that is, its supported media types.

The following table defines how the Express.js Router will expose

the API operations, and it should serve as a reference specification

for it:

Description

Sta

tus

Returns all the items in the catalog.

Int

ern

Ser

ver

Err

ryI

Returns all items for the selected category.

If the category does not exist, it returns 404.

OK,

404

Int

ern

Ser

ver

Err

/it

/{i

Returns a single item for the selected itemId.

If there is no such item, it returns 404.

OK,

404

Int

ern

Ser

ver

Err

/it

Creates a new item; if an item with the same

identifier exists, it will be updated. When an

item is created, a Location header is

returned. It provides the URL where the

newly created item can be accessed.

201

Cre

ate

Int

ern

Ser

ver

Err

/it

/{i

Updates an existing item; if an item with the

provided identifier does not exist, it creates

it. When an item is created, a Location

header is returned. It provides the URL

where the newly created item can be

accessed.

201

Cre

ate

Int

ern

Ser

ver

Err

/it

Deletes an existing item; if an item with the

OK,

404

/{i

provided identifier does not exist, it returns

404.

Int

ern

Ser

ver

Err

The catalog application handles two types of entities: items and

categories. Each item entity contains a collection of categories

where it belongs. As you can see, the category is just a logical entity

in our application; it will exist as long as there is at least one item

referencing it, and will cease to exist when no items refer it. This is

why the application exposes routes for exposing data manipulation

functions only for resources of type items, while the operations for

categories are more or less read only. Looking more carefully into

the URLs exposing the data manipulation operations for the items,

we can see a clear pattern aligning the URL to the REST

fundamental principles—a resource is exposed by a single URL, and

it supports resource manipulation actions that are determined by

the HTTP method of the request. To sum up, listed here are the

generally accepted rules that a well-defined API should follow. They

are semantically related to each resource manipulation operation:

When a new resource is created, the service makes use of

the 201 Created status code, followed by a location header

that specifies the URL where the newly created resource can

be accessed.

Operation that creates resources may be implemented to

gracefully reject creation of resources, which unique

identifiers already use; in such cases, the operation should

indicate a non-successful invocation with an appropriate

status code 409 Conflict, or a more general 400 BAD

REQUEST. However, a general status code should always

be followed by a meaningful explanation of what has gone

wrong. In our implementation, we choose a different

approach—we update the resource from the create

operation, if it exists, and notify the caller that the resource

was updated by returning the 200 OK status code instead

of 201 Created.

The Update operation resembles the create operation;

however, it always expects a resource identifier as a

parameter, if a resource with this identifier exists—it gets

updated with a new state provided in the body of the HTTP

PUT request. The 200 OK status code indicates successful

invocation. The implementation may decide to reject

handling of non-existent resources with the 404 Not

Found status code or creating a new resource with the

passed identifier. In that case, it would return the 201

Created status code, followed by a location header that

specifies the URL where the newly created resource can be

accessed. Our API makes use of the second option.

While successful deletion can be indicated with the 204

No Content status and further payload, most user agents

would expect the 2xx HTTP status to be followed by a body.

Thus, to stay compatible with most of the agents, our API

will indicate successful deletion with the 200 OK status

code, followed by a JSON payload:{'Status': 'Successfully

deleted'}. Status code 404 Not found will indicate that a

resource with the provided identifier does not exist.

As a general rule, 5XX should not indicate application state

errors but more severe errors, such as application server or

database failures.

It is best practice that update and create operations should

return as a payload to the entire state of the resource. For

instance, if a resource is created with a minimum set of

attributes, all non specified attribute will get default values;

the response body should contain the full state of the object.

The same is valid for updates; even if an update operation

updates the resource restate partially, the response should

return the complete state. This may save the user-agent an

additional GET request if they needed to check the new

state.

Now that we have defined some general recommendations on how

operations should behave, it's time to implement them in a new

version of the API.

Extensibility and versioning

We've already defined a few basic versioning rules in Chapter 3,

Building a Typical Web API. Let's apply them to the MongoDB

database-aware module we implemented in the previous chapter.

Our starting point would be to enable the current consumers of the

API to continue using the same version on a different URL. This will

keep them backward-compatible until they adopt and successfully

test the new version.

Keeping a REST API stable is not a question of only moving one

endpoint from one URI to another. It makes no sense to perform

redirection and afterward have an API that behaves differently.

Thus, we need to ensure that the behavior of the moved endpoint

stays the same. To ensure that we don't change the previously

implemented behavior, let's move the current behavior from

the catalog.js module to a new module by renaming the file to

catalogV1.js. Then, make a copy of it to the catalogV2.js module, where

we will introduce all new functionality; but before doing that, we

have to reroute Version 1 from /, /{categoryId}, /{itemId} to /v1,

/v1/{categoryId}, /v1/{itemId}:

const express = require('express');

const router = express.Router();

const catalogV1 = require('../modules/catalogV1');

const model = require('../model/item.js');

router.get('/v1/', function(request, response, next) {

catalogV1.findAllItems(response);

});

router.get('/v1/item/:itemId', function(request, response, next) {

console.log(request.url + ' : querying for ' + request.params.itemId);

catalogV1.findItemById(request.params.itemId, response);

});

router.get('/v1/:categoryId', function(request, response, next) {

console.log(request.url + ' : querying for ' + request.params.categoryId);

catalogV1.findItemsByCategory(request.params.categoryId, response);

});

router.post('/v1/', function(request, response, next) {

catalogV1.saveItem(request, response);

});

router.put('/v1/', function(request, response, next) {

catalogV1.saveItem(request, response);

});

router.delete('/v1/item/:itemId', function(request, response, next) {

catalogV1.remove(request, response);

});

router.get('/', function(request, response) {

console.log('Redirecting to v1');

response.writeHead(301, {'Location' : '/catalog/v1/'});

response.end('Version 1 is moved to /catalog/v1/: ');

});

module.exports = router;

Since Version 2 of our API is not yet implemented, executing a GET

request against / will result in receiving a 301 Moved Permanently HTTP

status, which will then redirect to /v1/. This will notify our

consumers that the API is evolving and that they will soon need to

decide whether to continue using Version 1 by explicitly requesting

its new URI or prepare for adopting Version 2.

Go ahead and give it a try! Start the modified node application and,

from Postman, make a GET request to http://localhost:3000/catalog:

You will see that your request is redirected to the newly routed

location at http://localhost:3000/catalog/v1.

Now that we have finalized Version 1 of the catalog, it's time to

think of further extensions that we can add in Version 2. Currently,

the catalog service supports listing of all items in a category and

fetching an item by its ID. It's about time to take full advantage of

MongoDB, being a document-oriented database, and implement a

function that will enable our API consumer to query for items based

on any of their attributes. For instance, list all items for a specific

category with an attribute that matches a query parameter, such as

price or color, or search by item name. RESTful services usually

expose document-oriented data. However, their usage is not limited

to documents only. In the next chapter, we will extend the catalog in

a way that it also stores binary data—an image that can be linked to

each item. For that purpose, we will use a MongoDB binary format

called Binary JSON (BSON) in the Working with arbitrary data

section in Chapter 6, Implementing a Full-Fledged RESTful Service.

Getting back to the searching extension, we've already used the

Mongoose.js model's find() and findOne() functions. So far, we used them

to provide the name of the document attribute to be searched with,

statically, in our JavaScript code. However, this filtering parameter

of find() is just a JSON object where the key is the document

attribute and the value is the attribute's value to be used in the

query. Here is the first new function we will add to Version 2. It

queries MongoDB by an arbitrary attribute and its value:

exports.findItemsByAttribute = function (key, value, response) {

var filter = {};

filter[key] = value;

CatalogItem.find(filter, function(error, result) {

if (error) {

console.error(error);

response.writeHead(500, contentTypePlainText);

response.end('Internal server error');

return;

} else {

if (!result) {

if (response != null) {

response.writeHead(200, contentTypeJson);

response.end({});

}

return;

}

if (response != null){

response.setHeader('Content-Type', 'application/json');

response.send(result);

}

});

}

This function calls find on the model with the provided attribute

and value as parameters. We will bind this function to the router's

/v2/item/ GET handler.

At the end, our aim is to have /v2/item/?currency=USD that returns only

records for items that are sold in USD currency, as indicated by the

value of the passed GET parameter. That way, if we modify the

model with additional attributes, such as color and size, we can

query for all items having the same color or any other attribute that

an item can have.

We will keep the old behavior of returning a list of all available

items when no parameters are provided within the query string, but

we will also parse the query string for the first provided GET

parameter and use it as a filter in the findItemsByAttribute() function:

router.get('/v2/items', function(request, response) {

var getParams = url.parse(request.url, true).query;

if (Object.keys(getParams).length == 0) {

catalogV2.findAllItems(response);

} else {

var key = Object.keys(getParams)[0];

var value = getParams[key];

catalogV2.findItemsByAttribute(key, value, response);

}

});

Perhaps the most interesting part in this function is the URL

parsing. As you can see, we keep using the same old strategy to

check whether any GET parameters are supplied. We parse the URL

in order to get the query string, and then we use the built-in

Object.keys function to check whether the parsed key/value list

contains elements. If it does, we take the first element and extract

its value. Both the key and the value are passed to the findByAttribute

function.

You may want to improve Version 2 further by providing search support by

multiple arguments that are provided by more than one GET parameters. I

will leave that to you as an exercise.

Discovering and exploring

RESTful services

The topic of discovering RESTful services has a long and

complicated history. The HTTP specification states that a resource

should be self-descriptive and that it should be identified uniquely

by a URI. Dependent resources should be linked by the dependency

using their own unique URIs. Discovering a RESTful service means

navigating from one service to another, following the links it

provides.

In the year 2009, a specification called Web Application

Discovery Language (WADL) was invented. It aims to

document every URI exposed from a web application, along with

the HTTP methods it supports and the parameter it expects. The

response media type of the URI is also described. This is very useful

for documenting purposes, and it's all that a WADL file can provide

us in terms of RESTful service provisioning.

Unfortunately, there is currently no Node.js module that can

automatically generate a WADL file for a given express route. We

will have to manually create a WADL file to demonstrate how it is

used by other clients for discovery.

The following listing shows a sample WADL file describing the

resources available at /catalog, /catalog/v2/{categoryId}:

<?xml version="1.0" encoding="UTF-8"?>

<application xmlns="http://wadl.dev.java.net/2009/02"

xmlns:service="http://localhost:8080/catalog/"

xmlns:xsd="http://www.w3.org/2001/XMLSchema">

</grammer>

</request>

<representation mediaType="application/xml"

element="service:item" />

</response>

<representation mediaType="text/plain" element="service:item"

</response>

</method>

</resource>

</request>

<representation mediaType="application/xml"

element="service:item" />

</response>

<representation mediaType="text/plain" element="service:item"

</response>

</method>

</resource>

</resources>

</application>

As you can see, the WADL format is very straightforward. It

basically describes the URI of each resource, providing information

about the media types it uses and the status codes that are expected

at that URI. Many third-party RESTful clients understand the

WADL language and can generate request messages out of a given

WADL file.

Let's import the WADL file in Postman. Click on the Import button

and select your WADL file:

Import a WADL in Postman to get a stub of the service. This is a screenshot for Postman. The individual settings

are not important here. The purpose of the image is just to show how the window looks.

As you can see, the result of importing the WADL file is that we

have a project ready to test each aspect of a REST service in the nick

of time. All the routes defined in the WADL file are now

conveniently available as separate request entities on the right

menu. That's not all; apart from the WADL standard, currently the

swagger documentation format is heavily adopted and has become

an informal standard for describing RESTful services, so we can

also use it to ease the adoption and discovery of our service. In the

next chapter, we will bind these description files to our service. This

is an important step in the phase of production preparation.

Linked data

Every catalog application supports an image or set of images bound

to this item. For that purpose, in the next chapter, we will see how

to work with binary objects in MongoDB. However, now is the time

to decide how to semantically link the binary data to an item

document. Extending the model schema in such a way that it

contains binary base64 representation of binary data in the

document is anything but a good idea, as mixing literally encoded

and binary data in one format is never a good idea. It increases the

complexity of the application and makes it error-prone:

{

"_id": "5a4c004b0eed73835833cc9a",

"itemId": "1",

"itemName": "Sports Watch",

"price": 100,

"currency": "EUR",

"categories": [

"Watches",

"Sports Watches"

"image":"

iVBORw0KGgoAAAANSUhEUgAAAJEAAACRCAMAAAD0BqoRAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJb

WFnZVJlYWR5ccllPAAAAyJpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj

0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0

iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuMC1jMDYwIDYxLjEzNDc3

NywgMjAxMC8wMi8xMi0xNzozMjowMCAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwO

i8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNuzjcmlwdGlv

biByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4b

Wxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RSZWY9Im

h0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiIHhtcDpDcmVhdG9

yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENTNSBNYWNpbnRvc2giIHhtcE1NOkluc3RhbmNlSUQ9Inht

cC5paWQ6MjMwNjQ1NDdFNjJCMTFERkI5QzU4OTFCMjJCQzEzM0EiIHhtcE1NOkRvY3VtZW50SUQ9I

nhtcC5kaWQ6MjMwNjQ1NDhFNjJCMTFERkI5QzU4OTFCMjJCQzEzM0EiPiA8eG1wTU06RGVyaXZlZE

Zyb20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDoyMzA2NDU0NUU2MkIxMURGQjlDNTg5MUIyMkJ

DMTMzQSIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDoyMzA2NDU0NkU2MkIxMURGQjlDNTg5MUIy

MkJDMTMzQSIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1wbWV0YT4gPD94c

GFja2V0IGVuZD0iciI/Px5Xq1XXhWFY1+v151/b3ij5tI/GPEVP0e8U/SPAABPLjHnaJ6XvAAAAAE

lFTkSuQmCC

Imagine how large a result for a non-filtered query can become just

for a few hundred items, if all of them had an image binary

representation as a value of a JSON attribute. To avoid that, we will

return the image for each item at a URL that is logically linked to

the resource's URL—/catalog/v2/item/{itemId}/image.

That way, if there is an image assigned to an item, it will be served

at a known location. This approach, however, does not semantically

link the binary item to its corresponding resource, as when

accessing the item at /catalog/v2/item/{itemId}, there would be no

indication of whether it has an image assigned or not. To solve this,

let's use a custom HTTP header in response of the item route:

GET http://localhost:3000/catalog/v2/item/1 HTTP/1.1

Host: localhost:3000

Connection: Keep-Alive

User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

HTTP/1.1 200 OK

X-Powered-By: Express

Content-Type: application/json; charset=utf-8

Content-Length: 152

Image-Url: http://localhost:3000/catalog/v2/item/1/image

ETag: W/"98-2nJj2mZdLV2YDME3WYCyEwIXfuA"

Date: Thu, 01 Feb 2018 13:50:43 GMT

Connection: keep-alive

{

"_id": "5a4c004b0eed73835833cc9a",

"itemId": "1",

"itemName": "Sports Watch",

"price": 100,

"currency": "EUR",

"__v": 0,

"categories": [

"Watches",

"Sports Watches"

]

}

When present in the response, the Image-Url header indicates that the

item has an additional resource bound to it, and the header value

provides the address where it is available. Using this approach, we

linked a binary resource semantically to our document.

In the next chapter, we will implement the routes that will handle

the manipulation of arbitrary items bound to the items in the

catalog.

Summary

In this chapter, we discussed in detail how resources should be

exposed via a RESTful API; we paid close attention to URL best

practices and looked into appropriate usage of the HTTP status

codes indicating each state of our operations.

We covered the topics of versioning and extensibility, where we

used the 301 Moved Permanently status code to automatically redirect API

calls to different URLs.

Finally, we figured out how to semantically link our resource items

to arbitrary binary represented data.

Implementing a Full Fledged

RESTful Service

So far, we have created a second version of our RESTful service, and

we had the two versions exposed by different URLs, ensuring

backward compatibility. We implemented unit tests for its database

layer and discussed how to use HTTP status codes appropriately. In

this chapter, we will extend that implementation—by providing

handling of non-document—binary data to the second version of

the service and linking it accordingly to the documents it relates to.

We will look at a convenient way of presenting large result sets to

consumers. For that purpose, we will introduce pagination as well

as further filtering capabilities to our API.

There are cases when caching data responses should be considered

as an option. We will look at its benefits and drawbacks and also

decide to enable caching when necessary.

Finally, we will dive into the discovery and exploration of REST

services.

To sum up, here's what should be further implemented to turn the

catalog data service into a full-fledged RESTful service:

Working with arbitrary data

Working with Linked data in the real world

Paging and filtering

Caching

Discovery and exploration

Working with arbitrary data

MongoDB utilizes BSON (Binary JSON) as the primary data format.

It is a binary format that stores key/value pairs in a single entity

called document. For example, a sample JSON, {"hello":"world"},

becomes \x16\x00\x00\x00\x02hello\x00\x06\x00\x00\x00world\x00\x00 when

encoded in BSON.

BSON stores data rather than literals. For instance, if an image is to

be part of the document, it will not have to be converted to a

base64-encoded string; instead, it will be directly stored as binary

data, unlike plain JSON, which will usually represent such data as

base64-encoded bytes, but that is obviously not the most efficient

way.

Mongoose schemas enable storing binary content in the BSON

format via the schema type—buffer. It stores binary content

(image, ZIP archive, and so on) up to 16 MB. The reason behind the

relatively small storage capacity is to prevent excessive usage of

memory and bandwidth during transmission.

The GridFS specification addresses this limitation of BSON and

enables you to work with data larger than 16 MB. GridFS divides

data into chunks stored as separate document entries. Each chunk,

by default, has a size of up to 255 KB. When data is requested from

the data store, the GridFS driver retrieves all the required chunks

and returns them in an assembled order, as if they had never been

divided. Not only does this mechanism allow storage of data larger

than 16 MB, it also enables consumers to retrieve data in portions

so that it doesn't have to be loaded completely into the memory.

Thus, the specification implicitly enables streaming support.

GridFS actually offers more—it supports storing metadata for the

given binary data, for example, its format, a filename, size, and so

on. The metadata is stored in a separate file and is available for

more complex queries. There is a very usable Node.js module called

gridfs-stream. It enables easy streaming of data in and out of

MongoDB, as on all other modules it is installed as an npm package.

So, let's install it globally and see how it is used; we will also use the

-s option to ensure that the dependencies in the

project's package.json are updated:

npm install -g -s gridfs-stream

To create a Grid instance, you are required to have a connection

opened to the database:

const mongoose = require('mongoose')

const Grid = require('gridfs-stream');

mongoose.connect('mongodb://localhost/catalog');

var connection = mongoose.connection;

var gfs = Grid(connection.db, mongoose.mongo);

Reading and writing into the stream is done through the

createReadStream() and createWriteStream() functions. Each piece of data

streamed into the database must have an ObjectId attribute set.

The ObjectId identifies binary data entry uniquely, just as it would

have identified any other document in MongoDB; using this ObjectId,

we can find or delete it from the MongoDB collection by this

identifier.

Let's extend the catalog service with functions for fetching, adding,

and deleting an image assigned to an item. For simplicity, the

service will support a single image per item, so there will be a single

function responsible for adding an image. It will overwrite an

existing image each time it is invoked, so an appropriate name for it

is saveImage:

exports.saveImage = function(gfs, request, response) {

var writeStream = gfs.createWriteStream({

filename : request.params.itemId,

mode : 'w'

});

writeStream.on('error', function(error) {

response.send('500', 'Internal Server Error');

console.log(error);

return;

})

writeStream.on('close', function() {

readImage(gfs, request, response);

});

request.pipe(writeStream);

}

As you can see, all we need to do to flush the data in MongoDB is to

create a GridFS write stream instance. It requires some options that

provide the ObjectId of the MongoDB entry and some additional

metadata, such as a title as well as the writing mode. Then, we

simply call the pipe function of the request. Piping will result in

flushing the data from the request to the write stream, and, in this

way, it will be safely stored in MongoDB. Once stored, the

close event associated with the writeStream will occur, and this is when

our function reads back whatever it has stored in the database and

returns that image in the HTTP response.

Retrieving an image is done the other way around—a readStream is

created with options, and the value of the _id parameter should

be the ObjectId of the arbitrary data, optional file name, and read

mode:

function readImage(gfs, request, response) {

var imageStream = gfs.createReadStream({

filename : request.params.itemId,

mode : 'r'

});

imageStream.on('error', function(error) {

console.log(error);

response.send('404', 'Not found');

return;

});

response.setHeader('Content-Type', 'image/jpeg');

imageStream.pipe(response);

}

Before piping the read stream to the response, the appropriate

Content-Type header has to be set so that the arbitrary data can be

presented to the client with an appropriate image media type,

image/jpeg, in our case.

Finally, we export from our module a function for fetching the

image back from MongoDB. We will use that function to bind it to

the express route that reads the image from the database:

exports.getImage = function(gfs, itemId, response) {

readImage(gfs, itemId, response);

};

Deleting arbitrary data from MongoDB is also straightforward. You

have to delete the entry from two internal MongoDB collections, the

fs.files, where all the files are kept, and from the fs.files.chunks:

exports.deleteImage = function(gfs, mongodb, itemId, response) {

console.log('Deleting image for itemId:' + itemId);

var options = {

filename : itemId,

};

var chunks = mongodb.collection('fs.files.chunks');

chunks.remove(options, function (error, image) {

if (error) {

console.log(error);

response.send('500', 'Internal Server Error');

return;

} else {

console.log('Successfully deleted image for item: ' + itemId);

}

});

var files = mongodb.collection('fs.files');

files.remove(options, function (error, image) {

if (error) {

console.log(error);

response.send('500', 'Internal Server Error');

return;

}

if (image === null) {

response.send('404', 'Not found');

return;

} else {

console.log('Successfully deleted image for primary item: ' +

itemId);

response.json({'deleted': true});

}

});

}

Let's bind the new functionality to the appropriate item route and

test it:

router.get('/v2/item/:itemId/image',

function(request, response){